HPCC SYSTEMS software Copyright (C) 2019 HPCC Systems.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+https://hpccsystems.com
You can disable some functionality in order to reduce the list of required components, if necessary. Optional components, such as the plugins for interfacing to external languages, will be disabled automatically if the required libraries and headers are not found at build time.
Optional dependencies:
To build with support for all the plugins for third party embedded code, additional dependencies may be required. If not found, the system will default to skipping those components.
From 7.4 the build system has been changed to make it easier to build on windows. It is dependent on two other projects chocolatey and vcpkg for installing the dependencies:
First insure that the R language is installed on your system. For Ubuntu use sudo apt-get install r-base-dev. For centos distributions use sudo yum install -y R-core-devel.
To install the prerequisites for building R support, use the following for all distros:
The minimum version of CMake required to build the HPCC Platform is 3.3.2 on Linux. You may need to download a recent version here at cmake.org.
Now you need to run CMake to populate your build directory with Makefiles and special configuration to build HPCC, packages, tests, etc.
A separate directory is required for the build files. In the examples below, the source directory is contained in ~/hpcc and the build directory is ~/hpcc/build. mkdir ~/hpcc/build
All cmake commands would normally need to be executed within the build directory: cd ~/hpcc/build
For release builds, do: cmake ../src
To enable a specific plugin in the build:
bash
cmake –D<Plugin Name>=ON ../src
+ make –j6 package
These are the current supported plugins:
CASSANDRAEMBED
REMBED
V8EMBED
MEMCACHED
PYEMBED
REDIS
JAVAEMBED
KAFKA
SQLITE3EMBED
MYSQLEMBED
If testing during development and you may want to include plugins (except R) in the package: cmake -DTEST_PLUGINS=ON ../src
To produce a debug build: cmake -DCMAKE_BUILD_TYPE:STRING=Debug ../src
To build the client tools only: cmake -DCLIENTTOOLS_ONLY=1 ../src
To enable signing of the ecl standard library, ensure you have a gpg private key loaded into your gpg keychain and do: # Add -DSIGN_MODULES_KEYID and -DSIGN_MODULES_PASSPHRASE if applicable cmake -DSIGN_MODULES=ON ../src
In some cases, users have found that when packaging for Ubuntu, the dpkg-shlibdeps portion of the packaging adds an exceptional amount of time to the build process. To turn this off (and to create a package without dynamic dependency generation) do: cmake -DUSE_SHLIBDEPS=OFF ../src
CMake will check for necessary dependencies like binutils, boost regex, cppunit, pthreads, etc. If everything is correct, it'll create the necessary Makefiles and you're ready to build HPCC.
We default to using libxslt in place of Xalan for xslt support. Should you prefer to use libxalan, you can specify -DUSE_LIBXALAN on the cmake command line.
The recommended method to install HPCC Systems on your machine (even for testing) is to use distro packages. CMake has already detected your system, so it know whether to generate TGZ files, DEB or RPM packages.
Just type:
make package
+
Alternatively you can use the build system agnostic variant:
cmake --build . --target package
+
and it will create the appropriate package for you to install. The package file will be created inside the build directory.
** Also make sure that bison is ahead of the system bison on your path. bison --version (The result should be > 2.4.1 )
** OS X has LDAP installed, but when compiling against it (/System/Library/Frameworks/LDAP.framework/Headers/ldap.h) you will get a #include nested too deeply, which is why you should install openldap.
Create a build directory - either as a child of hpcc or elsewhere
cd to the build directory
Use clang to build the clienttools (gcc4.2 cores when compiling some of the sources):
HPCC Systems offers an enterprise ready, open source supercomputing platform to solve big data problems. As compared to Hadoop, the platform offers analysis of big data using less code and less nodes for greater efficiencies and offers a single programming language, a single platform and a single architecture for efficient processing. HPCC Systems is a technology division of LexisNexis Risk Solutions.
In general, a new version of the HPCC Platform is released every 3 months. These releases can be either Major (with breaking changes) or Minor (with new features). Maintenance and security releases (point releases) are typically made weekly, and may occasionally include technical previews.
Maintenance releases are supported for the current and previous release, while security releases are supported for the current and previous two releases:
The HPCC Systems architecture incorporates the Thor and Roxie clusters as well as common middleware components, an external communications layer, client interfaces which provide both end-user services and system management tools, and auxiliary components to support monitoring and to facilitate loading and storing of filesystem data from external sources. An HPCC environment can include only Thor clusters, or both Thor and Roxie clusters. Each of these cluster types is described in more detail in the following sections below the architecture diagram.
Thor (the Data Refinery Cluster) is responsible for consuming vast amounts of data, transforming, linking and indexing that data. It functions as a distributed file system with parallel processing power spread across the nodes. A cluster can scale from a single node to thousands of nodes.
Single-threaded
Distributed parallel processing
Distributed file system
Powerful parallel processing programming language (ECL)
Optimized for Extraction, Transformation, Loading, Sorting, Indexing and Linking
Roxie (the Query Cluster) provides separate high-performance online query processing and data warehouse capabilities. Roxie (Rapid Online XML Inquiry Engine) is the data delivery engine used in HPCC to serve data quickly and can support many thousands of requests per node per second.
Multi-threaded
Distributed parallel processing
Distributed file system
Powerful parallel processing programming language (ECL)
cd /opt/HPCCSystems/testing/regress
+./ecl-test query --target thor nlppp.ecl
+
+
+
+
\ No newline at end of file
diff --git a/assets/BUILD_ME.md.Cs6Xx-2H.js b/assets/BUILD_ME.md.Cs6Xx-2H.js
new file mode 100644
index 00000000000..80b68823156
--- /dev/null
+++ b/assets/BUILD_ME.md.Cs6Xx-2H.js
@@ -0,0 +1,75 @@
+import{_ as a,c as s,a3 as i,o as n}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Build Instructions","description":"","frontmatter":{},"headers":[],"relativePath":"BUILD_ME.md","filePath":"BUILD_ME.md","lastUpdated":1731340314000}'),l={name:"BUILD_ME.md"};function t(o,e,p,r,d,c){return n(),s("div",null,e[0]||(e[0]=[i(`
HPCC SYSTEMS software Copyright (C) 2019 HPCC Systems.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+https://hpccsystems.com
You can disable some functionality in order to reduce the list of required components, if necessary. Optional components, such as the plugins for interfacing to external languages, will be disabled automatically if the required libraries and headers are not found at build time.
Optional dependencies:
To build with support for all the plugins for third party embedded code, additional dependencies may be required. If not found, the system will default to skipping those components.
From 7.4 the build system has been changed to make it easier to build on windows. It is dependent on two other projects chocolatey and vcpkg for installing the dependencies:
First insure that the R language is installed on your system. For Ubuntu use sudo apt-get install r-base-dev. For centos distributions use sudo yum install -y R-core-devel.
To install the prerequisites for building R support, use the following for all distros:
The minimum version of CMake required to build the HPCC Platform is 3.3.2 on Linux. You may need to download a recent version here at cmake.org.
Now you need to run CMake to populate your build directory with Makefiles and special configuration to build HPCC, packages, tests, etc.
A separate directory is required for the build files. In the examples below, the source directory is contained in ~/hpcc and the build directory is ~/hpcc/build. mkdir ~/hpcc/build
All cmake commands would normally need to be executed within the build directory: cd ~/hpcc/build
For release builds, do: cmake ../src
To enable a specific plugin in the build:
bash
cmake –D<Plugin Name>=ON ../src
+ make –j6 package
These are the current supported plugins:
CASSANDRAEMBED
REMBED
V8EMBED
MEMCACHED
PYEMBED
REDIS
JAVAEMBED
KAFKA
SQLITE3EMBED
MYSQLEMBED
If testing during development and you may want to include plugins (except R) in the package: cmake -DTEST_PLUGINS=ON ../src
To produce a debug build: cmake -DCMAKE_BUILD_TYPE:STRING=Debug ../src
To build the client tools only: cmake -DCLIENTTOOLS_ONLY=1 ../src
To enable signing of the ecl standard library, ensure you have a gpg private key loaded into your gpg keychain and do: # Add -DSIGN_MODULES_KEYID and -DSIGN_MODULES_PASSPHRASE if applicable cmake -DSIGN_MODULES=ON ../src
In some cases, users have found that when packaging for Ubuntu, the dpkg-shlibdeps portion of the packaging adds an exceptional amount of time to the build process. To turn this off (and to create a package without dynamic dependency generation) do: cmake -DUSE_SHLIBDEPS=OFF ../src
CMake will check for necessary dependencies like binutils, boost regex, cppunit, pthreads, etc. If everything is correct, it'll create the necessary Makefiles and you're ready to build HPCC.
We default to using libxslt in place of Xalan for xslt support. Should you prefer to use libxalan, you can specify -DUSE_LIBXALAN on the cmake command line.
The recommended method to install HPCC Systems on your machine (even for testing) is to use distro packages. CMake has already detected your system, so it know whether to generate TGZ files, DEB or RPM packages.
Just type:
make package
+
Alternatively you can use the build system agnostic variant:
cmake --build . --target package
+
and it will create the appropriate package for you to install. The package file will be created inside the build directory.
** Also make sure that bison is ahead of the system bison on your path. bison --version (The result should be > 2.4.1 )
** OS X has LDAP installed, but when compiling against it (/System/Library/Frameworks/LDAP.framework/Headers/ldap.h) you will get a #include nested too deeply, which is why you should install openldap.
Create a build directory - either as a child of hpcc or elsewhere
cd to the build directory
Use clang to build the clienttools (gcc4.2 cores when compiling some of the sources):
To build the makefiles just created above, run make
Executables will be created in ./<releasemode>/bin and ./<releasemode>/libs
To create a .dmg to install, run make package
`,110)]))}const b=a(l,[["render",t]]);export{u as __pageData,b as default};
diff --git a/assets/BUILD_ME.md.Cs6Xx-2H.lean.js b/assets/BUILD_ME.md.Cs6Xx-2H.lean.js
new file mode 100644
index 00000000000..80b68823156
--- /dev/null
+++ b/assets/BUILD_ME.md.Cs6Xx-2H.lean.js
@@ -0,0 +1,75 @@
+import{_ as a,c as s,a3 as i,o as n}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Build Instructions","description":"","frontmatter":{},"headers":[],"relativePath":"BUILD_ME.md","filePath":"BUILD_ME.md","lastUpdated":1731340314000}'),l={name:"BUILD_ME.md"};function t(o,e,p,r,d,c){return n(),s("div",null,e[0]||(e[0]=[i(`
HPCC SYSTEMS software Copyright (C) 2019 HPCC Systems.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+
+https://hpccsystems.com
You can disable some functionality in order to reduce the list of required components, if necessary. Optional components, such as the plugins for interfacing to external languages, will be disabled automatically if the required libraries and headers are not found at build time.
Optional dependencies:
To build with support for all the plugins for third party embedded code, additional dependencies may be required. If not found, the system will default to skipping those components.
From 7.4 the build system has been changed to make it easier to build on windows. It is dependent on two other projects chocolatey and vcpkg for installing the dependencies:
First insure that the R language is installed on your system. For Ubuntu use sudo apt-get install r-base-dev. For centos distributions use sudo yum install -y R-core-devel.
To install the prerequisites for building R support, use the following for all distros:
The minimum version of CMake required to build the HPCC Platform is 3.3.2 on Linux. You may need to download a recent version here at cmake.org.
Now you need to run CMake to populate your build directory with Makefiles and special configuration to build HPCC, packages, tests, etc.
A separate directory is required for the build files. In the examples below, the source directory is contained in ~/hpcc and the build directory is ~/hpcc/build. mkdir ~/hpcc/build
All cmake commands would normally need to be executed within the build directory: cd ~/hpcc/build
For release builds, do: cmake ../src
To enable a specific plugin in the build:
bash
cmake –D<Plugin Name>=ON ../src
+ make –j6 package
These are the current supported plugins:
CASSANDRAEMBED
REMBED
V8EMBED
MEMCACHED
PYEMBED
REDIS
JAVAEMBED
KAFKA
SQLITE3EMBED
MYSQLEMBED
If testing during development and you may want to include plugins (except R) in the package: cmake -DTEST_PLUGINS=ON ../src
To produce a debug build: cmake -DCMAKE_BUILD_TYPE:STRING=Debug ../src
To build the client tools only: cmake -DCLIENTTOOLS_ONLY=1 ../src
To enable signing of the ecl standard library, ensure you have a gpg private key loaded into your gpg keychain and do: # Add -DSIGN_MODULES_KEYID and -DSIGN_MODULES_PASSPHRASE if applicable cmake -DSIGN_MODULES=ON ../src
In some cases, users have found that when packaging for Ubuntu, the dpkg-shlibdeps portion of the packaging adds an exceptional amount of time to the build process. To turn this off (and to create a package without dynamic dependency generation) do: cmake -DUSE_SHLIBDEPS=OFF ../src
CMake will check for necessary dependencies like binutils, boost regex, cppunit, pthreads, etc. If everything is correct, it'll create the necessary Makefiles and you're ready to build HPCC.
We default to using libxslt in place of Xalan for xslt support. Should you prefer to use libxalan, you can specify -DUSE_LIBXALAN on the cmake command line.
The recommended method to install HPCC Systems on your machine (even for testing) is to use distro packages. CMake has already detected your system, so it know whether to generate TGZ files, DEB or RPM packages.
Just type:
make package
+
Alternatively you can use the build system agnostic variant:
cmake --build . --target package
+
and it will create the appropriate package for you to install. The package file will be created inside the build directory.
** Also make sure that bison is ahead of the system bison on your path. bison --version (The result should be > 2.4.1 )
** OS X has LDAP installed, but when compiling against it (/System/Library/Frameworks/LDAP.framework/Headers/ldap.h) you will get a #include nested too deeply, which is why you should install openldap.
Create a build directory - either as a child of hpcc or elsewhere
cd to the build directory
Use clang to build the clienttools (gcc4.2 cores when compiling some of the sources):
HPCC Systems offers an enterprise ready, open source supercomputing platform to solve big data problems. As compared to Hadoop, the platform offers analysis of big data using less code and less nodes for greater efficiencies and offers a single programming language, a single platform and a single architecture for efficient processing. HPCC Systems is a technology division of LexisNexis Risk Solutions.
In general, a new version of the HPCC Platform is released every 3 months. These releases can be either Major (with breaking changes) or Minor (with new features). Maintenance and security releases (point releases) are typically made weekly, and may occasionally include technical previews.
Maintenance releases are supported for the current and previous release, while security releases are supported for the current and previous two releases:
The HPCC Systems architecture incorporates the Thor and Roxie clusters as well as common middleware components, an external communications layer, client interfaces which provide both end-user services and system management tools, and auxiliary components to support monitoring and to facilitate loading and storing of filesystem data from external sources. An HPCC environment can include only Thor clusters, or both Thor and Roxie clusters. Each of these cluster types is described in more detail in the following sections below the architecture diagram.
Thor (the Data Refinery Cluster) is responsible for consuming vast amounts of data, transforming, linking and indexing that data. It functions as a distributed file system with parallel processing power spread across the nodes. A cluster can scale from a single node to thousands of nodes.
Single-threaded
Distributed parallel processing
Distributed file system
Powerful parallel processing programming language (ECL)
Optimized for Extraction, Transformation, Loading, Sorting, Indexing and Linking
Roxie (the Query Cluster) provides separate high-performance online query processing and data warehouse capabilities. Roxie (Rapid Online XML Inquiry Engine) is the data delivery engine used in HPCC to serve data quickly and can support many thousands of requests per node per second.
Multi-threaded
Distributed parallel processing
Distributed file system
Powerful parallel processing programming language (ECL)
cd /opt/HPCCSystems/testing/regress
+./ecl-test query --target thor nlppp.ecl
`,30)]))}const u=s(n,[["render",l]]);export{E as __pageData,u as default};
diff --git a/assets/README.md.aSZY6Tfx.lean.js b/assets/README.md.aSZY6Tfx.lean.js
new file mode 100644
index 00000000000..8317365e832
--- /dev/null
+++ b/assets/README.md.aSZY6Tfx.lean.js
@@ -0,0 +1,37 @@
+import{_ as s,c as a,a3 as i,o as t}from"./chunks/framework.DkhCEVKm.js";const E=JSON.parse('{"title":"Description / Rationale","description":"","frontmatter":{},"headers":[],"relativePath":"README.md","filePath":"README.md","lastUpdated":1731340314000}'),n={name:"README.md"};function l(r,e,o,p,h,c){return t(),a("div",null,e[0]||(e[0]=[i(`
HPCC Systems offers an enterprise ready, open source supercomputing platform to solve big data problems. As compared to Hadoop, the platform offers analysis of big data using less code and less nodes for greater efficiencies and offers a single programming language, a single platform and a single architecture for efficient processing. HPCC Systems is a technology division of LexisNexis Risk Solutions.
In general, a new version of the HPCC Platform is released every 3 months. These releases can be either Major (with breaking changes) or Minor (with new features). Maintenance and security releases (point releases) are typically made weekly, and may occasionally include technical previews.
Maintenance releases are supported for the current and previous release, while security releases are supported for the current and previous two releases:
The HPCC Systems architecture incorporates the Thor and Roxie clusters as well as common middleware components, an external communications layer, client interfaces which provide both end-user services and system management tools, and auxiliary components to support monitoring and to facilitate loading and storing of filesystem data from external sources. An HPCC environment can include only Thor clusters, or both Thor and Roxie clusters. Each of these cluster types is described in more detail in the following sections below the architecture diagram.
Thor (the Data Refinery Cluster) is responsible for consuming vast amounts of data, transforming, linking and indexing that data. It functions as a distributed file system with parallel processing power spread across the nodes. A cluster can scale from a single node to thousands of nodes.
Single-threaded
Distributed parallel processing
Distributed file system
Powerful parallel processing programming language (ECL)
Optimized for Extraction, Transformation, Loading, Sorting, Indexing and Linking
Roxie (the Query Cluster) provides separate high-performance online query processing and data warehouse capabilities. Roxie (Rapid Online XML Inquiry Engine) is the data delivery engine used in HPCC to serve data quickly and can support many thousands of requests per node per second.
Multi-threaded
Distributed parallel processing
Distributed file system
Powerful parallel processing programming language (ECL)
: - CMakeLists.txt - Root CMake file - version.cmake - common cmake file where version variables are set - build-config.h.cmake - cmake generation template for build-config.h
\\- cmake\\_modules/ - Directory storing modules and configurations for CMake
+
+: - FindXXXXX.cmake - CMake find files used to locate libraries,
+ headers, and binaries
+ - commonSetup.cmake - common configuration settings for the
+ entire project (contains configure time options)
+ - docMacros.cmake - common documentation macros used for
+ generating fop and pdf files
+ - optionDefaults.cmake - contains common variables for the
+ platform build
+ - distrocheck.sh - script that determines if the OS uses DEB
+ or RPM
+ - getpackagerevisionarch.sh - script that returns OS version
+ and arch in format used for packaging
+
+ \\- dependencies/ - Directory storing dependency files used for package dependencies
+
+ : - \\<OS\\>.cmake - File containing either DEB or RPM
+ dependencies for the given OS
+
+\\- build-utils/ - Directory for build related utilities
+
+: - cleanDeb.sh - script that unpacks a deb file and rebuilds
+ with fakeroot to clean up lintain errors/warnings
+
set_source_files_properties(<file> PROPERTIES GENERATED TRUE) - Must be set on generated source files or dependency generation fails and increases build time.
Using custom commands between multiple cmake files
GET_TARGET_PROPERTY(<VAR from other cmake file> <var for this file> LOCATION)
GET_TARGET_PROPERTY(ESDL_EXE esdl LOCATION) - will get from the top level cache the ESDL_EXE value and set it in esdl for your current cmake file
USE add_custom_command only when 100% needed.
All directories in a cmake project should have a CMakeLists.txt file and be called from the upper level project with an add_subdirectory or HPCC_ADD_SUBDIRECTORY
When you have a property that will be shared between cmake projects use define_property to set it in the top level cache.
define_property(GLOBAL PROPERTY TEST_TARGET BRIEF_DOCS "test doc" FULL_DOCS "Full test doc")
mark_as_advanced(TEST_TARGET) - this is required to force the property into the top level cache.CMake Layout:
NOT XXXXX_FOUND
+ Externals set
+ define needed vars for finding external based libraries/headers
+
+ Use Native set
+ use FIND_PATH to locate headers
+ use FIND_LIBRARY to find libs
+
+Include Cmake macros file for package handling
+define package handling args for find return (This will set XXXXX_FOUND)
+
+XXXXX_FOUND
+ perform any modifications you feel is needed for the find
+
+Mark defined variables used in package handling args as advanced for return
+
Will define when done:
XXXXX_FOUND
+XXXXX_INCLUDE_DIR
+XXXXX_LIBRARIES
+
(more can be defined, but must be at min the previous unless looking for only a binary)
For an example, see FindAPR.cmake
`,32)]))}const h=a(o,[["render",r]]);export{f as __pageData,h as default};
diff --git a/assets/cmake_modules_DOCUMENTATION.md.BLw9X_2f.lean.js b/assets/cmake_modules_DOCUMENTATION.md.BLw9X_2f.lean.js
new file mode 100644
index 00000000000..bb15406c97e
--- /dev/null
+++ b/assets/cmake_modules_DOCUMENTATION.md.BLw9X_2f.lean.js
@@ -0,0 +1,52 @@
+import{_ as a,c as i,a3 as t,o as n}from"./chunks/framework.DkhCEVKm.js";const f=JSON.parse('{"title":"CMake files structure and usage","description":"","frontmatter":{},"headers":[],"relativePath":"cmake_modules/DOCUMENTATION.md","filePath":"cmake_modules/DOCUMENTATION.md","lastUpdated":1731340314000}'),o={name:"cmake_modules/DOCUMENTATION.md"};function r(s,e,l,c,d,m){return n(),i("div",null,e[0]||(e[0]=[t(`
: - CMakeLists.txt - Root CMake file - version.cmake - common cmake file where version variables are set - build-config.h.cmake - cmake generation template for build-config.h
\\- cmake\\_modules/ - Directory storing modules and configurations for CMake
+
+: - FindXXXXX.cmake - CMake find files used to locate libraries,
+ headers, and binaries
+ - commonSetup.cmake - common configuration settings for the
+ entire project (contains configure time options)
+ - docMacros.cmake - common documentation macros used for
+ generating fop and pdf files
+ - optionDefaults.cmake - contains common variables for the
+ platform build
+ - distrocheck.sh - script that determines if the OS uses DEB
+ or RPM
+ - getpackagerevisionarch.sh - script that returns OS version
+ and arch in format used for packaging
+
+ \\- dependencies/ - Directory storing dependency files used for package dependencies
+
+ : - \\<OS\\>.cmake - File containing either DEB or RPM
+ dependencies for the given OS
+
+\\- build-utils/ - Directory for build related utilities
+
+: - cleanDeb.sh - script that unpacks a deb file and rebuilds
+ with fakeroot to clean up lintain errors/warnings
+
set_source_files_properties(<file> PROPERTIES GENERATED TRUE) - Must be set on generated source files or dependency generation fails and increases build time.
Using custom commands between multiple cmake files
GET_TARGET_PROPERTY(<VAR from other cmake file> <var for this file> LOCATION)
GET_TARGET_PROPERTY(ESDL_EXE esdl LOCATION) - will get from the top level cache the ESDL_EXE value and set it in esdl for your current cmake file
USE add_custom_command only when 100% needed.
All directories in a cmake project should have a CMakeLists.txt file and be called from the upper level project with an add_subdirectory or HPCC_ADD_SUBDIRECTORY
When you have a property that will be shared between cmake projects use define_property to set it in the top level cache.
define_property(GLOBAL PROPERTY TEST_TARGET BRIEF_DOCS "test doc" FULL_DOCS "Full test doc")
mark_as_advanced(TEST_TARGET) - this is required to force the property into the top level cache.CMake Layout:
NOT XXXXX_FOUND
+ Externals set
+ define needed vars for finding external based libraries/headers
+
+ Use Native set
+ use FIND_PATH to locate headers
+ use FIND_LIBRARY to find libs
+
+Include Cmake macros file for package handling
+define package handling args for find return (This will set XXXXX_FOUND)
+
+XXXXX_FOUND
+ perform any modifications you feel is needed for the find
+
+Mark defined variables used in package handling args as advanced for return
+
Will define when done:
XXXXX_FOUND
+XXXXX_INCLUDE_DIR
+XXXXX_LIBRARIES
+
(more can be defined, but must be at min the previous unless looking for only a binary)
For an example, see FindAPR.cmake
`,32)]))}const h=a(o,[["render",r]]);export{f as __pageData,h as default};
diff --git a/assets/devdoc_CodeGenerator.md.DjbPmndU.js b/assets/devdoc_CodeGenerator.md.DjbPmndU.js
new file mode 100644
index 00000000000..abcb9a68e71
--- /dev/null
+++ b/assets/devdoc_CodeGenerator.md.DjbPmndU.js
@@ -0,0 +1,13 @@
+import{_ as t,c as a,a3 as i,o}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Eclcc/Code generator","description":"","frontmatter":{"title":"Eclcc/Code generator"},"headers":[],"relativePath":"devdoc/CodeGenerator.md","filePath":"devdoc/CodeGenerator.md","lastUpdated":1731340314000}'),r={name:"devdoc/CodeGenerator.md"};function s(n,e,l,h,d,c){return o(),a("div",null,e[0]||(e[0]=[i(`
The code generator has to do its job accurately. If the code generator does not correctly map the ECL to the workunit it can lead to corrupt data and invalid results. Problems like that can often be very hard and frustrating for the ECL users to track down.
There is also a strong emphasis on generating output that is as good as possible. Eclcc contains many different optimization stages, and is extensible to allow others to be easily added.
Eclcc needs to be able to cope with reasonably large jobs. Queries that contain several megabytes of ECL, and generate tens of thousands of activities, and 10s of Mb of C++ are routine. These queries need to be processed relatively quickly.
Nearly all the processing of ECL is done using an expression graph. The representation of the expression graph has some particular characteristics:
Once the nodes in the expression graph have been created they are NEVER modified.
Nodes in the expression graph are ALWAYS commoned up if they are identical.
Each node in the expression graph is link counted (see below) to track its lifetime.
If a modified graph is required a new graph is created (sharing nodes from the old one)
The ECL language is a declarative language, and in general is assumed to be pure - i.e. there are no side-effects, expressions can be evaluated lazily and re-evaluating an expression causes no problems. This allows eclcc to transform the graph in lots of interesting ways. (Life is never that simple so there are mechanisms for handling the exceptions.)
One of the main challenges with eclcc is converting the declarative ECL code into imperative C++ code. One key problem is it needs to try to ensure that code is only evaluated when it is required, but that it is also only evaluated once. It isn't always possible to satisfy both constraints - for example a global dataset expression used within a child query. Should it be evaluated once before the activity containing the child query is called, or each time the child query is called? If it is called on demand then it may not be evaluated as efficiently...
This issue complicates many of the optimizations and transformations that are done to the queries. Long term the plan is to allow the engines to support more delayed lazy-evaluation, so that whether something is evaluated is more dynamic rather than static.
The idealised view of the processing within eclcc follows the following stages:
Parse the ECL into an expression graph.
Expand out function calls.
Normalize the expression graph so it is in a consistent format.
Normalize the references to fields within datasets to tie them up with their scopes.
Apply various global optimizations.
Translate logical operations into the activities that will implement them.
Resource and generate the global graph
For each activity, resource, optimize and generate its child graphs.
In practice the progression is not so clear cut. There tends to be some overlap between the different stages, and some of them may occur in slightly different orders. However the order broadly holds.
Before any change is accepted for the code generator it is always run against several regression suites to ensure that it doesn't introduce any problems, and that the change has the desired effect. There are several different regression suites:
testing/regress/ecl - The run time regression suite.
ecl/regress - a compiler regression suite. This contains tests that cannot run and error tests.
LN private suite - This contains a large selection (>10Gb) of archived queries. The contain proprietary code so unfortunately cannot be released as open source.
The ecl/regress directory contains a script 'regress.sh' that is used for running the regression tests. It should be executed in the directory containing the ecl files. The script generates the c++ code (and workunits) for each of the source files to a target directory, and then executes a comparison program to compare the new results with a previous "golden" reference set.
Before making any changes to the compiler, a reference set should be created by running the regression script and copying the generated files to the reference directory.
(A version of this command resides in a shell script in each of my regression suite directories, with the -t and -c options adapted for each suite.)
For a full list of options execute the script with no parameters, or take a look at the script itself. A couple of useful options are:
The script can be run on a single file by using the -q option.
The (-e) option selects the path of the eclcc. This is particularly useful when running from the build directory (see below), or using multiple build directories to compare behaviour between different versions.
We strongly recommend using a comparison program which allows rules to be defined to ignore certain differences (e.g., beyond compare).
It is much quicker to run eclcc directly from the build directory, rather than deploying a system and running eclcc from there. To do this you need to configure some options that eclcc requires, e.g. where the include files are found. The options can be set by either setting environment variables or by specifiying options in an eclcc.ini file. The following are the names of the different options:
The eclcc.ini can either be a file in the local directory, or specified on the eclcc command line with -specs. Including the settings in a local eclcc.ini file also it easy to debug eclcc directly from the build directory within the eclipse environment.
There is an option for eclcc to output a logging file, and another to specify the level of detail in that logging file. If the detail level is above 500 then the expresssion tree for the query is output to the logging file after each of the code transformations. The tracing is very useful for tracking down at which stage inconsistencies are introduced in the expression graph, and also for learning how each transformation affects the query.
The output format defaults to ECL - which is regenerated from the expression tree. (This ECL cannot generally be compiled without editing - partly because it contains extra annoations.) Use either of the following:
There is a debug option (-ftraceIR) that generates an intermediate representation of the expression graph rather than regenerating ECL. The output tends to be less compact and harder to read quickly, but has the advantage of being better structured, and contains more details of the internal representation. ecl/hql/hqlir.cpp contains more details of the format.
Adding extra logging into the source code
If you want to add tracing of expressions at any point in the code generation then adding either of the following calls will include the expression details in the log file:
dbglogExpr(expr); // regenerate the ecl for an expression. See other functions in ecl/hql/hqlthql.hpp
EclIR::dbglogIR(expr); // regenerate the IR for an expression. See other functions in ecl/hql/hqlir.hpp
Logging while debugging
If you are debugging inside gdb it is often useful to be able to dump out details of an expression. Calling EclIR:dump_ir(expr); will generate the IR to stdout.
p EclIR::dump_ir(expr)
The function can also be used with multiple parameters. Each expression will be dumped out, but common child nodes will only be generated once. This can be very useful when trying to determine the difference between two expressions. The quickest way is to call EclIR::dump_ir(expr1, expr2). The first difference between the expressions will be the expression that follows the first "return".
Expression sequence ids.
Sometimes it can be hard to determine where a particular IHqlExpression node was created. If that is the case, then defining DEBUG_TRACK_INSTANCEID (in ecl/hql/hqlexpr.ipp) will add a unique sequence number to each IHqlExpression that is created. There is also a function checkSeqId() at the start of ecl/hql/hqlexpr.cpp which is called whenever an expression is created, linked, released etc.. Setting a breakpoint in that function can allow you to trace back exactly when and why a particular node was created.
The key data structure within eclcc is the graph representation. The design has some key elements.
Once a node is created it is never modified.
Some derived information (e.g., sort order, number of records, unique hash, ...) might be calculated and stored in the class after it has been created, but that doesn't change what the node represents in any way. Some nodes are created in stages - e.g., records, modules. These nodes are marked as fully completed when closeExpr() is called, after which they cannot be modified.
Nodes are always commoned up.
If the same operator has the same arguments and type then there will be a unique IHqlExpression to represent it. This helps ensure that graphs stay as graphs and don't get converted to trees. It also helps with optimizations, and allows code duplicated in two different contexts to be brought together.
The nodes are link counted.
Link counts are used to control the lifetime of the expression objects. Whenever a reference to an expression node is held, its link count is increased, and decreased when no longer required. The node is freed when there are no more references. (This generally works well, but does give us problems with forward references.)
The access to the graph is through interfaces.
The main interfaces are IHqlExpression, IHqlDataset and IHqlScope. They are all defined in hqlexpr.hpp. The aim of the interfaces is to hide the implementation of the expression nodes so they can be restructured and changed without affecting any other code.
The expression classes use interfaces and a type field rather than polymorphism. This could be argued to be bad object design...but.
There are more than 500 different possible operators. If a class was created for each of them the system would quickly become unwieldy. Instead there are several different classes which model the different types of expression (dataset/expression/scope).
The interfaces contain everything needed to create and interrogate an expression tree, but they do not contain functionality for directly processing the graph.
To avoid some of the shortcomings of type fields there are various mechanisms for accessing derived attributes which avoid interrogating the type field.
Memory consumption is critical.
It is not unusual to have 10M or even 100M nodes in memory as a query is being processed. At that scale the memory consumption of each node matters - so great care should be taken when considering increasing the size of the objects. The node classes contain a class hierarchy which is there purely to reduce the memory consumption - not to reflect the functionality. With no memory constraints they wouldn't be there, but removing a single pointer per node can save 1Gb of memory usage for very complex queries.
This is the interface that is used to walk and interrogate the expression graph once it has been created. Some of the main functions are: getOperator() What does this node represent? It returns a member of the node_operator enumerated type. numChildren() How many arguments does node have? queryChild(unsigned n) What is the nth child? If the argument is out of range it returns NULL. queryType() The type of this node. queryBody() Used to skip annotations (see below) queryProperty() Does this node have a child which is an attribute that matches a given name. (see below for more about attributes). queryValue() For a no_constant return the value of the constant. It returns NULL otherwise.
The nodes in the expression graph are created through factory functions. Some of the expression types have specialised functions - e.g., createDataset, createRow, createDictionary, but scalar expressions and actions are normally created with createValue().
Note: Generally ownership of the arguments to the createX() functions are assumed to be taken over by the newly created node.
The values of the enumeration constants in node_operator are used to calculate "crcs" which are used to check if the ECL for a query matches, and if disk and index record formats match. It contains quite a few legacy entries no_unusedXXX which can be used for new operators (otherwise new operators must be added to the end).
This interface is implemented by records, and is used to map names to the fields within the records. If a record contains IFBLOCKs then each of the fields in the ifblock is defined in the IHqlSimpleScope for the containing record.
Normally obtained by calling IHqlExpression::queryScope(). It is primarily used in the parser to resolve fields from within modules.
The ECL is parsed on demand so as the symbol is looked up it may cause a cascade of ECL to be compiled. The lookup context (HqlLookupContext ) is passed to IHqlScope::lookupSymbol() for several reasons:
It contains information about the active repository - the source of the ECL which will be dynamically parsed.
It contains caches of expanded functions - to avoid repeating expansion transforms.
Some members are used for tracking definitions that are read to build dependency graphs, or archives of submitted queries.
The interface IHqlScope currently has some members that are used for creation; this should be refactored and placed in a different interface.
This is normally obtained by calling IHqlExpression::queryDataset(). It has shrunk in size over time, and could quite possibly be folded into IHqlExpression with little pain.
There is a distinction in the code generator between "tables" and "datasets". A table (IHqlDataset::queryTable()) is a dataset operation that defines a new output record. Any operation that has a transform or record that defines an output record (e.g., PROJECT,TABLE) is a table, whilst those that don't (e.g., a filter, dedup) are not. There are a few apparent exceptions -e.g., IF (This is controlled by definesColumnList() which returns true the operator is a table.)
There are two related by slightly different concepts. An attribute refers to the explicit flags that are added to operators (e.g., , LOCAL, KEEP(n) etc. specified in the ECL or some internal attributes added by the code generator). There are a couple of different functions for creating attributes. createExtraAttribute() should be used by default. createAttribute() is reserved for an attribute that never has any arguments, or in unusual situations where it is important that the arguments are never transformed. They are tested using queryAttribute()/hasAttribute() and represented by nodes of kind no_attr/no_expr_attr.
The term "property" refers to computed information (e.g., record counts) that can be derived from the operator, its arguments and attributes. They are covered in more detail below.
Fields can be selected from active rows of a dataset in three main ways:
Some operators define LEFT/RIGHT to represent an input or processed dataset. Fields from these active rows are referenced with LEFT.<field-name>. Here LEFT or RIGHT is the "selector".
Other operators use the input dataset as the selector. E.g., myFile(myFile.id != 0). Here the input dataset is the "selector".
Often when the input dataset is used as the selector it can be omitted. E.g., myFile(id != 0). This is implicitly expanded by the PARSER to the second form. A reference to a field is always represented in the expression graph as a node of kind no_select (with createSelectExpr). The first child is the selector, and the second is the field. Needless to say there are some complications...
LEFT/RIGHT.
The problem is that the different uses of LEFT/RIGHT need to be disambiguated since there may be several different uses of LEFT in a query. This is especially true when operations are executed in child queries. LEFT is represented by a node no_left(record, selSeq). Often the record is sufficient to disambiguate the uses, but there are situations where it isn't enough. So in addition no_left has a child which is a selSeq (selector sequence) which is added as a child attribute of the PROJECT or other operator. At parse time it is a function of the input dataset that is later normalized to a unique id to reduce the transformation work.
Active datasets. It is slightly more complicated - because the dataset used as the selector can be any upstream dataset up to the nearest table. So the following ECL code is legal:
Here the reference to x.id in the definition of z is referring to a field in the input dataset.
Because of these semantics the selector in a normalized tree is actually inputDataset->queryNormalizedSelector() rather than inputDatset. This function currently returns the table expression node (but it may change in the future see below).
In some situations ECL allows child datasets to be treated as a dataset without an explicit NORMALIZE. E.g., EXISTS(myDataset.ChildDataset);
This is primarily to enable efficient aggregates on disk files to be generated, but it adds some complications with an expression of the form dataset.childdataset.grandchild. E.g.,:
In the first example dataset.childdataset within the dataset.childdataset.grandchild is a reference to a dataset that doesn't have an active cursor and needs to be iterated), whilst in the second it refers to an active cursor.
To differentiate between the two, all references to fields within datasets/rows that don't have active selectors have an additional attribute("new") as a child of the select. So a no_select with a "new" attribute requires the dataset to be created, one without is a member of an active dataset cursor.
If you have a nested row, the new attribute is added to the selection from the dataset, rather than the selection from the nested row. The functions queryDatasetCursor() and querySelectorDataset()) are used to help interpret the meaning.
(An alternative would be to use a different node from no_select - possibly this should be considered - it would be more space efficient.)
The expression graph generated by the ECL parser doesn't contain any new attributes. These are added as one of the first stages of normalizing the expression graph. Any code that works on normalized expressions needs to take care to interpret no_selects correctly.
When an expression graph is transformed and none of the records are changed, the representation of LEFT/RIGHT remains the same. This means any no_select nodes in the expression tree will also stay the same.
However, if the transform modifies a table (highly likely) it means that the selector for the second form of field selector will also change. Unfortunately this means that transforms often cannot be short-circuited.
It could significantly reduce the extent of the graph that needs traversing, and the number of nodes replaced in a transformed graph if this could be avoided. One possibility is to use a different value for dataset->queryNormalizedSelector() using a unique id associated with the table. I think it would be a good long term change, but it would require unique ids (similar to the selSeq) to be added to all table expressions, and correctly preserved by any optimization.
Sometimes it is useful to add information into the expression graph (e.g., symbol names, position information) that doesn't change the meaning, but should be preserved. Annotations allow information to be added in this way.
An annotation's implementation of IHqlExpression generally delegates the majority of the methods through to the annotated expression. This means that most code that interrogates the expression graph can ignore their presence, which simplifies the caller significantly. However transforms need to be careful (see below).
Information about the annotation can be obtained by calling IHqlExpression:: getAnnotationKind() and IHqlExpression:: queryAnnotation().
In legacy ECL you will see code like the following::
EXPORT a(x) := FUNCTION
+ Y := F(x);
+ OUTPUT(Y);
+ RETURN G(Y);
+END;
+
The assumption is that whenever a(x) is evaluated the value of Y will be output. However that doesn't particularly fit in with a declarative expression graph. The code generator creates a special node (no_compound) with child(0) as the output action, and child(1) as the value to be evaluated (g(Y)).
If the expression ends up being included in the final query then the action will also be included (via the no_compound). At a later stage the action is migrated to a position in the graph where actions are normally evaluated.
There are many pieces of information that it is useful to know about a node in the expression graph - many of which would be expensive to recomputed each time there were required. Eclcc has several mechanisms for caching derived information so it is available efficiently.
Boolean flags - getInfoFlags()/getInfoFlags2().
There are many Boolean attributes of an expression that are useful to know - e.g., is it constant, does it have side-effects, does it reference any fields from a dataset etc. etc. The bulk of these are calculated and stored in a couple of members of the expression class. They are normally retrieved via accessor functions e.g., containsAssertKeyed(IHqlExpression*).
Active datasets - gatherTablesUsed().
It is very common to want to know which datasets an expression references. This information is calculated and cached on demand and accessed via the IHqlExpression::gatherTablesUsed() functions. There are a couple of other functions IHqlExpression::isIndependentOfScope() and IHqlExpression::usesSelector() which provide efficient functions for common uses.
Information stored in the type.
Currently datasets contain information about sort order, distribution and grouping as part of the expression type. This information should be accessed through the accessor functions applied to the expression (e.g., isGrouped(expr)). At some point in the future it is planned to move this information as a general derived property (see next).
Other derived property.
There is a mechanism (in hqlattr) for calculating and caching an arbitrary derived property of an expression. It is currently used for number of rows, location-independent representation, maximum record size etc. . There are typically accessor functions to access the cached information (rather than calling the underlying IHqlExpression::queryAttribute() function).
Helper functions.
Some information doesn't need to be cached because it isn't expensive to calculate, but rather than duplicating the code, a helper function is provided. E.g., queryOriginalRecord() and hasUnknownTransform(). They are not part of the interface because the number would make the interface unwieldy and they can be completely calculated from the public functions.
However, it can be very hard to find the function you are looking for, and they would greatly benefit from being grouped e.g., into namespaces.
One of the key processes in eclcc is walking and transforming the expression graphs. Both of these are covered by the term transformations. One of the key things to bear in mind is that you need to walk the expression graph as a graph, not as a tree. If you have already examined a node once you shouldn't repeat the work - otherwise the execution time may be exponential with node depth.
Other things to bear in mind
If a node isn't modified don't create a new one - return a link to the old one.
You generally need to walk the graph and gather some information before creating a modified graph. Sometimes creating a new graph can be short-circuited if no changes will be required.
Sometimes you can be tempted to try and short-circuit transforming part of a graph (e.g., the arguments to a dataset activity), but because of the way references to fields within dataset work that often doesn't work.
If an expression is moved to another place in the graph, you need to be very careful to check if the original context was conditional and that the new context is not.
The meaning of expressions can be context dependent. E.g., References to active datasets can be ambiguous.
Never walk the expressions as a tree, always as a graph!
Be careful with annotations.
It is essential that an expression that is used in different contexts with different annotations (e.g., two different named symbols) is consistently transformed. Otherwise it is possible for a graph to be converted into a tree. E.g.,:
A := x; B := x; C = A + B;
+
must not be converted to:
A' := x'; B' := X''; C' := A' + B';
+
For this reason most transformers will check if expr->queryBody() matches expr, and if not will transform the body (the unannotated expression), and then clone any annotations.
Some examples of the work done by transformations are:
Constant folding.
Expanding function calls.
Walking the graph and reporting warnings.
Optimizing the order and removing redundant activities.
Reducing the fields flowing through the generated graph.
Spotting common sub expressions.
Calculating the best location to evaluate an expression (e.g., globally instead of in a child query).
Many, many others.
Some more details on the individual transforms are given below..
The first job of eclcc is to parse the ECL into an expression graph. The source for the ECL can come from various different sources (archive, source files, remote repository). The details are hidden behind the IEclSource/IEclSourceCollection interfaces. The createRepository() function is then used to resolve and parse the various source files on demand.
Several things occur while the ECL is being parsed:
Function definitions are expanded inline.
A slightly unusual behaviour. It means that the expression tree is a fully expanded expression -which is better suited to processing and optimizing.
Some limited constant folding occurs.
When a function is expanded, often it means that some of the test conditions are always true/false. To reduce the transformations the condition may be folded early on.
When a symbol is referenced from another module this will recursively cause the ECL for that module (or definition within that module) to be parsed.
Currently the semantic checking is done as the ECL is parsed.
If we are going to fully support template functions and delayed expansion of functions this will probably have to change so that a syntax tree is built first, and then the semantic checking is done later.
There are various problems with the expression graph that comes out of the parser:
Records can have values as children (e.g., { myField := infield.value} ), but it causes chaos if record definitions can change while other transformations are going on. So the normalization removes values from fields.
Some activities use records to define the values that output records should contain (e.g., TABLE). These are now converted to another form (e.g., no_newusertable).
Sometimes expressions have multiple definition names. Symbols and annotations are rationalized and commoned up to aid commoning up other expressions.
Some PATTERN definitions are recursive by name. They are resolved to a form that works if all symbols are removed.
The CASE/MAP representation for a dataset/action is awkward for the transforms to process. They are converted to nested Ifs.
(At some point a different representation might be a good idea.)
EVALUATE is a weird syntax. Instances are replaced with equivalent code which is much easier to subsequently process.
The datasets used in index definitions are primarily there to provide details of the fields. The dataset itself may be very complex and may not actually be used. The dataset input to an index is replaced with a dummy "null" dataset to avoid unnecessary graph transforming, and avoid introducing any additional incorrect dependencies.
Generally if you use LEFT/RIGHT then the input rows are going to be available wherever they are used. However if they are passed into a function, and that function uses them inside a definition marked as global then that is invalid (since by definition global expressions don't have any context).
Similarly if you use syntax <dataset>.<field>, its validity and meaning depends on whether <dataset> is active. The scope transformer ensures that all references to fields are legal, and adds a "new" attribute to any no_selects where it is necessary.
This transform simplifies the expression tree. Its aim is to simplify scalar expressions, and dataset expressions that are valid whether or not the nodes are shared. Some examples are:
1 + 2 => 3 and any other operation on scalar constants.
IF(true, x, y) => x
COUNT(<empty-dataset>) => 0
IF (a = b, 'c', 'd') = 'd' => IF(a=b, false, true) => a != b
Simplifying sorts, projects filters on empty datasets
Most of the optimizations are fairly standard, but a few have been added to cover more esoteric examples which have occurred in queries over the years.
This transform also supports the option to percolate constants through the graph. E.g., if a project assigns the value 3 to a field, it can substitute the value 3 wherever that field is used in subsequent activities. This can often lead to further opportunities for constant folding (and removing fields in the implicit project).
This transformer is used to simplify, combine and reorder dataset expressions. The transformer takes care to count the number of times each expression is used to ensure that none of the transformations cause duplication. E.g., swapping a filter with a sort is a good idea, but if there are two filters of the same sort and they are both swapped you will now be duplicating the sort.
ECL tends to be written as general purpose definitions which can then be combined. This can lead to potential inefficiencies - e.g., one definition may summarise some data in 20 different ways, this is then used by another definition which only uses a subset of those results. The implicit project transformer tracks the data flow at each point through the expression graph, and removes any fields that are not required.
This often works in combination with the other optimizations. For instance the constant percolation can remove the need for fields, and removing fields can sometimes allow a left outer join to be converted to a project.
is this the correct term? Should it be a query? This should really be independent of this document...)
The code generator ultimately creates workunits. A workunit completely describes a generated query. It consists of two parts. There is an xml component - this contains the workflow information, the various execution graphs, and information about options. It also describes which inputs can be supplied to the query and what results are generated. The other part is the generated shared object compiled from the generated C++. This contains functions and classes that are used by the engines to execute the queries. Often the xml is compressed and stored as a resource within the shared object -so the shared object contains a complete workunit.
The actions in a workunit are divided up into individual workflow items. Details of when each workflow item is executed, what its dependencies are stored in the <Workflow> section of the xml. The generated code also contains a class definition, with a method perform() which is used to execute the actions associated with a particular workflow item. (The class instances are created by calling the exported createProcess() factory function).
The generated code for an individual workflow item will typically call back into the engine at some point to execute a graph.
The activity graphs are stored in the xml. The graph contains details of which activities are required, how those activities link together, what dependencies there are between the activities. For each activity it might contain the following information:
A unique id.
The "kind" of the activity (from enum ThorActivityKind in eclhelper.hpp)
The ECL that created the activity.
Name of the original definition
Location (e.g., file, line number) of the original ECL.
Information about the record size, number of rows, sort order etc.
Hints which control options for a particular activity (e.g,, the number of threads to use while sorting).
Record counts and stats once the job has executed.
Each activity in a graph also has a corresponding helper class instance in the generated code. (The name of the class is cAc followed by the activity number, and the exported factory method is fAc followed by the activity number.) These classes implement the interfaces defined in eclhelper.hpp.
The engine uses the information from the xml to produce a graph of activities that need to be executed. It has a general purpose implementation of each activity kind, and it uses the class instance to tailor that general activity to the specific use e.g., what is the filter condition, what fields are set up, what is the sort order?
The workunit xml contains details of what inputs can be supplied when that workunit is run. These correspond to STORED definitions in the ECL. The result xml also contains the schema for the results that the workunit will generate.
Once an instance of the workunit has been run, the values of the results may be written back into dali's copy of the workunit so they can be retrieved and displayed.
Compile time is an issue - especially for small on-demand queries. To help reduce compile times (and dependencies with the rest of the system) the number of header files included by the generated code is kept to a minimum. In particular references to jlib, boost and icu are kept within the implementation of the runtime functions, and are not included in the public dependencies.
Thread-safe.
It should be possible to use the members of an activity helper from multiple threads without issue. The helpers may contain some context dependent state, so different instances of the helpers are needed for concurrent use from different contexts (e.g., expansions of a graph.)
Concise.
The code should be broadly readable, but the variable names etc. are chosen to generate compact code.
Functional.
Generally the generated code assigns to a variable once, and doesn't modify it afterwards. Some assignments may be conditional, but once the variable is evaluated it isn't updated. (There are of course a few exceptions - e.g., dataset iterators)
First a few pointers to help understand the code within eclcc:
It makes extensive use of link counting. You need understand that concept to get very far.
If something is done more than once then that is generally split into a helper function.
The helper functions aren't generally added to the corresponding interface (e.g., IHqlExpression) because the interface would become bloated. Instead they are added as global functions. The big disadvantage of this approach is they can be hard to find. Even better would be for them to be rationalised and organised into namespaces.
The code is generally thread-safe unless there would be a significant performance implication. In generally all the code used by the parser for creating expressions is thread safe. Expression graph transforms are thread-safe, and can execute in parallel if a constant (NUM_PARALLEL_TRANSFORMS) is increased. The data structures used to represent the generated code are NOT thread-safe.
Much of the code generation is structured fairly procedurally, with classes used to process the stages within it.
There is a giant "God" class HqlCppTranslator - which could really do with refactoring.
The eclcc parser uses the standard tools bison and flex to process the ECL and convert it to a
: expression graph. There are a couple of idiosyncrasies with the way it is implemented.
Macros with fully qualified scope.
Slightly unusually macros are defined in the same way that other definitions are - in particular to can have references to macros in other modules. This means that there are references to macros within the grammar file (instead of being purely handled by a pre-processor). It also means the lexer keeps an active stack of macros being processed.
Attributes on operators.
Many of the operators have optional attributes (e.g., KEEP, INNER, LOCAL, ...). If these were all reserved words it would remove a significant number of keywords from use as symbols, and could also mean that when a new attribute was added it broke existing code. To avoid this the lexer looks ahead in the parser tables (by following the potential reductions) to see if the token really could come next. If it can't then it isn't reserved as a symbol.
As the workunit is created the code generator builds up the generated code and the xml for the workunit. Most of the xml generation is encapsulated within the IWorkUnit interface. The xml for the graphs is created in an IPropertyTree, and added to the workunit as a block.
The C++ generation is ultimately controlled by some template files (thortpl.cpp). The templates are plain text and contain references to allow named sections of code to be expanded at particular points.
The code generator builds up some structures in memory for each of those named sections. Once the generation is complete some peephole optimization is applied to the code. This structure is walked to expand each named section of code as required.
The BuildCtx class provides a cursor into that generated C++. It will either be created for a given named section, or more typically from another BuildCtx. It has methods for adding the different types of statements. Some are simple (e.g., addExpr()), whilst some create a compound statement (e.g., addFilter). The compound statements change the active selector so any new statements are added within that compound statement.
As well as building up a tree of expressions, this data structure also maintains a tree of associations. For instance when a value is evaluated and assigned to a temporary variable, the logical value is associated with that temporary. If the same expression is required later, the association is matched, and the temporary value is used instead of recalculating it. The associations are also used to track the active datasets, classes generated for row-meta information, activity classes etc. etc.
Each activity in an expression graph will have an associated class generated in the C++. Each different activity kind expects a helper that implements a particular IHThorArg interface. E.g., a sort activity of kind TAKsort requires a helper that implements IHThorSortArg. The associated factory function is used to create instances of the helper class.
The generated class might take one of two forms:
A parameterised version of a library class. These are generated for simple helpers that don't have many variations (e.g., CLibrarySplitArg for TAKsplit), or for special cases that occur very frequently (CLibraryWorkUnitReadArg for internal results).
A class derived from a skeleton implementation of that helper (typically CThorXYZ implementing interface IHThorXYZ). The base class has default implementations of some of the functions, and any exceptions are implemented in the derived class.
This is a class that is used by the engines to encapsulate all the information about a single row -e.g., the format that each activity generates. It is an implementation of the IOutputMeta interface. It includes functions to
The same expression nodes are used for representing expressions in the generated C++ as the original ECL expression graph. It is important to keep track of whether an expression represents untranslated ECL, or the "translated" C++. For instance ECL has 1 based indexes, while C++ is zero based. If you processed the expression x[1] it might get translated to x[0] in C++. Translating it again would incorrectly refer to x[-1].
There are two key classes used while building the C++ for an ECL expression:
CHqlBoundExpr.
This represents a value that has been converted to C++. Depending on the type, one or more of the fields will be filled in.
CHqlBoundTarget.
This represents the target of an assignment -C++ variable(s) that are going to be assigned the result of evaluating an expression. It is almost always passed as a const parameter to a function because the target is well-defined and the function needs to update that target.
A C++ expression is sometimes converted back to an ECL pseudo-expression by calling getTranslatedExpr(). This creates an expression node of kind no_translated to indicate the child expression has already been converted.
The generation code for expressions has a hierarchy of calls. Each function is there to allow optimal code to be generated - e.g., not creating a temporary variable if none are required. A typical flow might be:
buildExpr(ctx, expr, bound).
Evaluate the ecl expression "expr" and save the C++ representation in the class bound. This might then call through to...
buildTempExpr(ctx, expr, bound);
Create a temporary variable, and evaluate expr and assign it to that temporary variable.... Which then calls.
buildExprAssign(ctx, target, expr);
evaluate the expression, and ensure it is assigned to the C++ target "target".
The default implementation might be to call buildExpr....
An operator must either be implemented in buildExpr() (calling a function called doBuildExprXXX) or in buildExprAssign() (calling a function called doBuildAssignXXX). Some operators are implemented in both places if there are different implementations that would be more efficient in each context.
Similarly there are several different assignment functions:
buildAssign(ctx, <ecl-target>, <ecl-value>);
buildExprAssign(ctx, <c++-target>, <ecl-value>);
assign(ctx, <C++target>, <c++source>)
The different varieties are there depending on whether the source value or targets have already been translated. (The names could be rationalised!)
Most dataset operations are only implemented as activities (e.g., PARSE, DEDUP). If these are used within a transform/filter then eclcc will generate a call to a child query. An activity helper for the appropriate operation will then be generated.
However a subset of the dataset operations can also be evaluated inline without calling a child query. Some examples are filters, projects, and simple aggregation. It removes the overhead of the child query call in the simple cases, and often generates more concise code.
When datasets are evaluated inline there is a similar hierarchy of function calls:
buildDatasetAssign(ctx, target, expr);
Evaluate the dataset expression, and assign it to the target (a builder interface). This may then call....
buildIterate(ctx, expr)
Iterate through each of the rows in the dataset expression in turn. Which may then call...
buildDataset(ctx, expr, target, format)
Build the entire dataset, and return it as a single value.
Some of the operations (e.g., aggregating a filtered dataset) can be done more efficiently by summing and filtering an iterator, than forcing the filtered dataset to be evaluated first.
The interface IHqlCppDatasetCursor allows the code generator to iterate through a dataset, or select a particular element from a dataset. It is used to hide the different representation of datasets, e.g.,
Blocked - the rows are in a contiguous block of memory appended one after another.
Array - the dataset is represented by an array of pointers to the individual rows.
Link counted - similar to array, but each element is also link counted.
Nested. Sometimes the cursor may iterate through multiple levels of child datasets.
Generally rows that are serialized (e.g., on disk) are in blocked format, and they are stored as link counted rows in memory.
The IReferenceSelector interface and the classes in hqltcppc[2] provide an interface for getting and setting values within a row of a dataset. They hide the details of the layout - e.g., csv/xml/raw data, and the details of exactly how each type is represented in the row.
The current implementation of keys in HPCC uses a format which uses a separate 8 byte integer field which was historically used to store the file position in the original file. Other complications are that the integer fields are stored big-endian, and signed integer values are biased.
This introduces some complication in the way indexes are handled. You will often find that the logical index definition is replaced with a physical index definition, followed by a project to convert it to the logical view. A similar process occurs for disk files to support VIRTUAL(FILEPOSITION) etc.
As mentioned at the start of this document, one of the main challenges with eclcc is converting the declarative ECL code into imperative C++ code. The direction we are heading in is to allow the engines to support more lazy-evaluation so possibly in this instance to evaluate it the first time it is used (although that may potentially be much less efficient). This will allow the code generator to relax some of its current assumptions.
There are several example queries which are already producing pathological behaviour from eclcc, causing it to generate C++ functions which are many thousands of lines long.
Currently the grammar for the parser is too specialised. In particular the separate productions for expression, datasets, actions cause problems - e.g., it is impossible to properly allow sets of datasets to be treated in the same way as other sets.
The semantic checking (and probably semantic interpretation) is done too early. Really the parser should build up a syntax tree, and then disambiguate it and perform the semantic checks on the syntax tree.
The function calls should probably be expanded later than they are. I have tried in the past and hit problems, but I can't remember all the details. Some are related to the semantic checking.
`,223)]))}const f=t(r,[["render",s]]);export{u as __pageData,f as default};
diff --git a/assets/devdoc_CodeGenerator.md.DjbPmndU.lean.js b/assets/devdoc_CodeGenerator.md.DjbPmndU.lean.js
new file mode 100644
index 00000000000..abcb9a68e71
--- /dev/null
+++ b/assets/devdoc_CodeGenerator.md.DjbPmndU.lean.js
@@ -0,0 +1,13 @@
+import{_ as t,c as a,a3 as i,o}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Eclcc/Code generator","description":"","frontmatter":{"title":"Eclcc/Code generator"},"headers":[],"relativePath":"devdoc/CodeGenerator.md","filePath":"devdoc/CodeGenerator.md","lastUpdated":1731340314000}'),r={name:"devdoc/CodeGenerator.md"};function s(n,e,l,h,d,c){return o(),a("div",null,e[0]||(e[0]=[i(`
The code generator has to do its job accurately. If the code generator does not correctly map the ECL to the workunit it can lead to corrupt data and invalid results. Problems like that can often be very hard and frustrating for the ECL users to track down.
There is also a strong emphasis on generating output that is as good as possible. Eclcc contains many different optimization stages, and is extensible to allow others to be easily added.
Eclcc needs to be able to cope with reasonably large jobs. Queries that contain several megabytes of ECL, and generate tens of thousands of activities, and 10s of Mb of C++ are routine. These queries need to be processed relatively quickly.
Nearly all the processing of ECL is done using an expression graph. The representation of the expression graph has some particular characteristics:
Once the nodes in the expression graph have been created they are NEVER modified.
Nodes in the expression graph are ALWAYS commoned up if they are identical.
Each node in the expression graph is link counted (see below) to track its lifetime.
If a modified graph is required a new graph is created (sharing nodes from the old one)
The ECL language is a declarative language, and in general is assumed to be pure - i.e. there are no side-effects, expressions can be evaluated lazily and re-evaluating an expression causes no problems. This allows eclcc to transform the graph in lots of interesting ways. (Life is never that simple so there are mechanisms for handling the exceptions.)
One of the main challenges with eclcc is converting the declarative ECL code into imperative C++ code. One key problem is it needs to try to ensure that code is only evaluated when it is required, but that it is also only evaluated once. It isn't always possible to satisfy both constraints - for example a global dataset expression used within a child query. Should it be evaluated once before the activity containing the child query is called, or each time the child query is called? If it is called on demand then it may not be evaluated as efficiently...
This issue complicates many of the optimizations and transformations that are done to the queries. Long term the plan is to allow the engines to support more delayed lazy-evaluation, so that whether something is evaluated is more dynamic rather than static.
The idealised view of the processing within eclcc follows the following stages:
Parse the ECL into an expression graph.
Expand out function calls.
Normalize the expression graph so it is in a consistent format.
Normalize the references to fields within datasets to tie them up with their scopes.
Apply various global optimizations.
Translate logical operations into the activities that will implement them.
Resource and generate the global graph
For each activity, resource, optimize and generate its child graphs.
In practice the progression is not so clear cut. There tends to be some overlap between the different stages, and some of them may occur in slightly different orders. However the order broadly holds.
Before any change is accepted for the code generator it is always run against several regression suites to ensure that it doesn't introduce any problems, and that the change has the desired effect. There are several different regression suites:
testing/regress/ecl - The run time regression suite.
ecl/regress - a compiler regression suite. This contains tests that cannot run and error tests.
LN private suite - This contains a large selection (>10Gb) of archived queries. The contain proprietary code so unfortunately cannot be released as open source.
The ecl/regress directory contains a script 'regress.sh' that is used for running the regression tests. It should be executed in the directory containing the ecl files. The script generates the c++ code (and workunits) for each of the source files to a target directory, and then executes a comparison program to compare the new results with a previous "golden" reference set.
Before making any changes to the compiler, a reference set should be created by running the regression script and copying the generated files to the reference directory.
(A version of this command resides in a shell script in each of my regression suite directories, with the -t and -c options adapted for each suite.)
For a full list of options execute the script with no parameters, or take a look at the script itself. A couple of useful options are:
The script can be run on a single file by using the -q option.
The (-e) option selects the path of the eclcc. This is particularly useful when running from the build directory (see below), or using multiple build directories to compare behaviour between different versions.
We strongly recommend using a comparison program which allows rules to be defined to ignore certain differences (e.g., beyond compare).
It is much quicker to run eclcc directly from the build directory, rather than deploying a system and running eclcc from there. To do this you need to configure some options that eclcc requires, e.g. where the include files are found. The options can be set by either setting environment variables or by specifiying options in an eclcc.ini file. The following are the names of the different options:
The eclcc.ini can either be a file in the local directory, or specified on the eclcc command line with -specs. Including the settings in a local eclcc.ini file also it easy to debug eclcc directly from the build directory within the eclipse environment.
There is an option for eclcc to output a logging file, and another to specify the level of detail in that logging file. If the detail level is above 500 then the expresssion tree for the query is output to the logging file after each of the code transformations. The tracing is very useful for tracking down at which stage inconsistencies are introduced in the expression graph, and also for learning how each transformation affects the query.
The output format defaults to ECL - which is regenerated from the expression tree. (This ECL cannot generally be compiled without editing - partly because it contains extra annoations.) Use either of the following:
There is a debug option (-ftraceIR) that generates an intermediate representation of the expression graph rather than regenerating ECL. The output tends to be less compact and harder to read quickly, but has the advantage of being better structured, and contains more details of the internal representation. ecl/hql/hqlir.cpp contains more details of the format.
Adding extra logging into the source code
If you want to add tracing of expressions at any point in the code generation then adding either of the following calls will include the expression details in the log file:
dbglogExpr(expr); // regenerate the ecl for an expression. See other functions in ecl/hql/hqlthql.hpp
EclIR::dbglogIR(expr); // regenerate the IR for an expression. See other functions in ecl/hql/hqlir.hpp
Logging while debugging
If you are debugging inside gdb it is often useful to be able to dump out details of an expression. Calling EclIR:dump_ir(expr); will generate the IR to stdout.
p EclIR::dump_ir(expr)
The function can also be used with multiple parameters. Each expression will be dumped out, but common child nodes will only be generated once. This can be very useful when trying to determine the difference between two expressions. The quickest way is to call EclIR::dump_ir(expr1, expr2). The first difference between the expressions will be the expression that follows the first "return".
Expression sequence ids.
Sometimes it can be hard to determine where a particular IHqlExpression node was created. If that is the case, then defining DEBUG_TRACK_INSTANCEID (in ecl/hql/hqlexpr.ipp) will add a unique sequence number to each IHqlExpression that is created. There is also a function checkSeqId() at the start of ecl/hql/hqlexpr.cpp which is called whenever an expression is created, linked, released etc.. Setting a breakpoint in that function can allow you to trace back exactly when and why a particular node was created.
The key data structure within eclcc is the graph representation. The design has some key elements.
Once a node is created it is never modified.
Some derived information (e.g., sort order, number of records, unique hash, ...) might be calculated and stored in the class after it has been created, but that doesn't change what the node represents in any way. Some nodes are created in stages - e.g., records, modules. These nodes are marked as fully completed when closeExpr() is called, after which they cannot be modified.
Nodes are always commoned up.
If the same operator has the same arguments and type then there will be a unique IHqlExpression to represent it. This helps ensure that graphs stay as graphs and don't get converted to trees. It also helps with optimizations, and allows code duplicated in two different contexts to be brought together.
The nodes are link counted.
Link counts are used to control the lifetime of the expression objects. Whenever a reference to an expression node is held, its link count is increased, and decreased when no longer required. The node is freed when there are no more references. (This generally works well, but does give us problems with forward references.)
The access to the graph is through interfaces.
The main interfaces are IHqlExpression, IHqlDataset and IHqlScope. They are all defined in hqlexpr.hpp. The aim of the interfaces is to hide the implementation of the expression nodes so they can be restructured and changed without affecting any other code.
The expression classes use interfaces and a type field rather than polymorphism. This could be argued to be bad object design...but.
There are more than 500 different possible operators. If a class was created for each of them the system would quickly become unwieldy. Instead there are several different classes which model the different types of expression (dataset/expression/scope).
The interfaces contain everything needed to create and interrogate an expression tree, but they do not contain functionality for directly processing the graph.
To avoid some of the shortcomings of type fields there are various mechanisms for accessing derived attributes which avoid interrogating the type field.
Memory consumption is critical.
It is not unusual to have 10M or even 100M nodes in memory as a query is being processed. At that scale the memory consumption of each node matters - so great care should be taken when considering increasing the size of the objects. The node classes contain a class hierarchy which is there purely to reduce the memory consumption - not to reflect the functionality. With no memory constraints they wouldn't be there, but removing a single pointer per node can save 1Gb of memory usage for very complex queries.
This is the interface that is used to walk and interrogate the expression graph once it has been created. Some of the main functions are: getOperator() What does this node represent? It returns a member of the node_operator enumerated type. numChildren() How many arguments does node have? queryChild(unsigned n) What is the nth child? If the argument is out of range it returns NULL. queryType() The type of this node. queryBody() Used to skip annotations (see below) queryProperty() Does this node have a child which is an attribute that matches a given name. (see below for more about attributes). queryValue() For a no_constant return the value of the constant. It returns NULL otherwise.
The nodes in the expression graph are created through factory functions. Some of the expression types have specialised functions - e.g., createDataset, createRow, createDictionary, but scalar expressions and actions are normally created with createValue().
Note: Generally ownership of the arguments to the createX() functions are assumed to be taken over by the newly created node.
The values of the enumeration constants in node_operator are used to calculate "crcs" which are used to check if the ECL for a query matches, and if disk and index record formats match. It contains quite a few legacy entries no_unusedXXX which can be used for new operators (otherwise new operators must be added to the end).
This interface is implemented by records, and is used to map names to the fields within the records. If a record contains IFBLOCKs then each of the fields in the ifblock is defined in the IHqlSimpleScope for the containing record.
Normally obtained by calling IHqlExpression::queryScope(). It is primarily used in the parser to resolve fields from within modules.
The ECL is parsed on demand so as the symbol is looked up it may cause a cascade of ECL to be compiled. The lookup context (HqlLookupContext ) is passed to IHqlScope::lookupSymbol() for several reasons:
It contains information about the active repository - the source of the ECL which will be dynamically parsed.
It contains caches of expanded functions - to avoid repeating expansion transforms.
Some members are used for tracking definitions that are read to build dependency graphs, or archives of submitted queries.
The interface IHqlScope currently has some members that are used for creation; this should be refactored and placed in a different interface.
This is normally obtained by calling IHqlExpression::queryDataset(). It has shrunk in size over time, and could quite possibly be folded into IHqlExpression with little pain.
There is a distinction in the code generator between "tables" and "datasets". A table (IHqlDataset::queryTable()) is a dataset operation that defines a new output record. Any operation that has a transform or record that defines an output record (e.g., PROJECT,TABLE) is a table, whilst those that don't (e.g., a filter, dedup) are not. There are a few apparent exceptions -e.g., IF (This is controlled by definesColumnList() which returns true the operator is a table.)
There are two related by slightly different concepts. An attribute refers to the explicit flags that are added to operators (e.g., , LOCAL, KEEP(n) etc. specified in the ECL or some internal attributes added by the code generator). There are a couple of different functions for creating attributes. createExtraAttribute() should be used by default. createAttribute() is reserved for an attribute that never has any arguments, or in unusual situations where it is important that the arguments are never transformed. They are tested using queryAttribute()/hasAttribute() and represented by nodes of kind no_attr/no_expr_attr.
The term "property" refers to computed information (e.g., record counts) that can be derived from the operator, its arguments and attributes. They are covered in more detail below.
Fields can be selected from active rows of a dataset in three main ways:
Some operators define LEFT/RIGHT to represent an input or processed dataset. Fields from these active rows are referenced with LEFT.<field-name>. Here LEFT or RIGHT is the "selector".
Other operators use the input dataset as the selector. E.g., myFile(myFile.id != 0). Here the input dataset is the "selector".
Often when the input dataset is used as the selector it can be omitted. E.g., myFile(id != 0). This is implicitly expanded by the PARSER to the second form. A reference to a field is always represented in the expression graph as a node of kind no_select (with createSelectExpr). The first child is the selector, and the second is the field. Needless to say there are some complications...
LEFT/RIGHT.
The problem is that the different uses of LEFT/RIGHT need to be disambiguated since there may be several different uses of LEFT in a query. This is especially true when operations are executed in child queries. LEFT is represented by a node no_left(record, selSeq). Often the record is sufficient to disambiguate the uses, but there are situations where it isn't enough. So in addition no_left has a child which is a selSeq (selector sequence) which is added as a child attribute of the PROJECT or other operator. At parse time it is a function of the input dataset that is later normalized to a unique id to reduce the transformation work.
Active datasets. It is slightly more complicated - because the dataset used as the selector can be any upstream dataset up to the nearest table. So the following ECL code is legal:
Here the reference to x.id in the definition of z is referring to a field in the input dataset.
Because of these semantics the selector in a normalized tree is actually inputDataset->queryNormalizedSelector() rather than inputDatset. This function currently returns the table expression node (but it may change in the future see below).
In some situations ECL allows child datasets to be treated as a dataset without an explicit NORMALIZE. E.g., EXISTS(myDataset.ChildDataset);
This is primarily to enable efficient aggregates on disk files to be generated, but it adds some complications with an expression of the form dataset.childdataset.grandchild. E.g.,:
In the first example dataset.childdataset within the dataset.childdataset.grandchild is a reference to a dataset that doesn't have an active cursor and needs to be iterated), whilst in the second it refers to an active cursor.
To differentiate between the two, all references to fields within datasets/rows that don't have active selectors have an additional attribute("new") as a child of the select. So a no_select with a "new" attribute requires the dataset to be created, one without is a member of an active dataset cursor.
If you have a nested row, the new attribute is added to the selection from the dataset, rather than the selection from the nested row. The functions queryDatasetCursor() and querySelectorDataset()) are used to help interpret the meaning.
(An alternative would be to use a different node from no_select - possibly this should be considered - it would be more space efficient.)
The expression graph generated by the ECL parser doesn't contain any new attributes. These are added as one of the first stages of normalizing the expression graph. Any code that works on normalized expressions needs to take care to interpret no_selects correctly.
When an expression graph is transformed and none of the records are changed, the representation of LEFT/RIGHT remains the same. This means any no_select nodes in the expression tree will also stay the same.
However, if the transform modifies a table (highly likely) it means that the selector for the second form of field selector will also change. Unfortunately this means that transforms often cannot be short-circuited.
It could significantly reduce the extent of the graph that needs traversing, and the number of nodes replaced in a transformed graph if this could be avoided. One possibility is to use a different value for dataset->queryNormalizedSelector() using a unique id associated with the table. I think it would be a good long term change, but it would require unique ids (similar to the selSeq) to be added to all table expressions, and correctly preserved by any optimization.
Sometimes it is useful to add information into the expression graph (e.g., symbol names, position information) that doesn't change the meaning, but should be preserved. Annotations allow information to be added in this way.
An annotation's implementation of IHqlExpression generally delegates the majority of the methods through to the annotated expression. This means that most code that interrogates the expression graph can ignore their presence, which simplifies the caller significantly. However transforms need to be careful (see below).
Information about the annotation can be obtained by calling IHqlExpression:: getAnnotationKind() and IHqlExpression:: queryAnnotation().
In legacy ECL you will see code like the following::
EXPORT a(x) := FUNCTION
+ Y := F(x);
+ OUTPUT(Y);
+ RETURN G(Y);
+END;
+
The assumption is that whenever a(x) is evaluated the value of Y will be output. However that doesn't particularly fit in with a declarative expression graph. The code generator creates a special node (no_compound) with child(0) as the output action, and child(1) as the value to be evaluated (g(Y)).
If the expression ends up being included in the final query then the action will also be included (via the no_compound). At a later stage the action is migrated to a position in the graph where actions are normally evaluated.
There are many pieces of information that it is useful to know about a node in the expression graph - many of which would be expensive to recomputed each time there were required. Eclcc has several mechanisms for caching derived information so it is available efficiently.
Boolean flags - getInfoFlags()/getInfoFlags2().
There are many Boolean attributes of an expression that are useful to know - e.g., is it constant, does it have side-effects, does it reference any fields from a dataset etc. etc. The bulk of these are calculated and stored in a couple of members of the expression class. They are normally retrieved via accessor functions e.g., containsAssertKeyed(IHqlExpression*).
Active datasets - gatherTablesUsed().
It is very common to want to know which datasets an expression references. This information is calculated and cached on demand and accessed via the IHqlExpression::gatherTablesUsed() functions. There are a couple of other functions IHqlExpression::isIndependentOfScope() and IHqlExpression::usesSelector() which provide efficient functions for common uses.
Information stored in the type.
Currently datasets contain information about sort order, distribution and grouping as part of the expression type. This information should be accessed through the accessor functions applied to the expression (e.g., isGrouped(expr)). At some point in the future it is planned to move this information as a general derived property (see next).
Other derived property.
There is a mechanism (in hqlattr) for calculating and caching an arbitrary derived property of an expression. It is currently used for number of rows, location-independent representation, maximum record size etc. . There are typically accessor functions to access the cached information (rather than calling the underlying IHqlExpression::queryAttribute() function).
Helper functions.
Some information doesn't need to be cached because it isn't expensive to calculate, but rather than duplicating the code, a helper function is provided. E.g., queryOriginalRecord() and hasUnknownTransform(). They are not part of the interface because the number would make the interface unwieldy and they can be completely calculated from the public functions.
However, it can be very hard to find the function you are looking for, and they would greatly benefit from being grouped e.g., into namespaces.
One of the key processes in eclcc is walking and transforming the expression graphs. Both of these are covered by the term transformations. One of the key things to bear in mind is that you need to walk the expression graph as a graph, not as a tree. If you have already examined a node once you shouldn't repeat the work - otherwise the execution time may be exponential with node depth.
Other things to bear in mind
If a node isn't modified don't create a new one - return a link to the old one.
You generally need to walk the graph and gather some information before creating a modified graph. Sometimes creating a new graph can be short-circuited if no changes will be required.
Sometimes you can be tempted to try and short-circuit transforming part of a graph (e.g., the arguments to a dataset activity), but because of the way references to fields within dataset work that often doesn't work.
If an expression is moved to another place in the graph, you need to be very careful to check if the original context was conditional and that the new context is not.
The meaning of expressions can be context dependent. E.g., References to active datasets can be ambiguous.
Never walk the expressions as a tree, always as a graph!
Be careful with annotations.
It is essential that an expression that is used in different contexts with different annotations (e.g., two different named symbols) is consistently transformed. Otherwise it is possible for a graph to be converted into a tree. E.g.,:
A := x; B := x; C = A + B;
+
must not be converted to:
A' := x'; B' := X''; C' := A' + B';
+
For this reason most transformers will check if expr->queryBody() matches expr, and if not will transform the body (the unannotated expression), and then clone any annotations.
Some examples of the work done by transformations are:
Constant folding.
Expanding function calls.
Walking the graph and reporting warnings.
Optimizing the order and removing redundant activities.
Reducing the fields flowing through the generated graph.
Spotting common sub expressions.
Calculating the best location to evaluate an expression (e.g., globally instead of in a child query).
Many, many others.
Some more details on the individual transforms are given below..
The first job of eclcc is to parse the ECL into an expression graph. The source for the ECL can come from various different sources (archive, source files, remote repository). The details are hidden behind the IEclSource/IEclSourceCollection interfaces. The createRepository() function is then used to resolve and parse the various source files on demand.
Several things occur while the ECL is being parsed:
Function definitions are expanded inline.
A slightly unusual behaviour. It means that the expression tree is a fully expanded expression -which is better suited to processing and optimizing.
Some limited constant folding occurs.
When a function is expanded, often it means that some of the test conditions are always true/false. To reduce the transformations the condition may be folded early on.
When a symbol is referenced from another module this will recursively cause the ECL for that module (or definition within that module) to be parsed.
Currently the semantic checking is done as the ECL is parsed.
If we are going to fully support template functions and delayed expansion of functions this will probably have to change so that a syntax tree is built first, and then the semantic checking is done later.
There are various problems with the expression graph that comes out of the parser:
Records can have values as children (e.g., { myField := infield.value} ), but it causes chaos if record definitions can change while other transformations are going on. So the normalization removes values from fields.
Some activities use records to define the values that output records should contain (e.g., TABLE). These are now converted to another form (e.g., no_newusertable).
Sometimes expressions have multiple definition names. Symbols and annotations are rationalized and commoned up to aid commoning up other expressions.
Some PATTERN definitions are recursive by name. They are resolved to a form that works if all symbols are removed.
The CASE/MAP representation for a dataset/action is awkward for the transforms to process. They are converted to nested Ifs.
(At some point a different representation might be a good idea.)
EVALUATE is a weird syntax. Instances are replaced with equivalent code which is much easier to subsequently process.
The datasets used in index definitions are primarily there to provide details of the fields. The dataset itself may be very complex and may not actually be used. The dataset input to an index is replaced with a dummy "null" dataset to avoid unnecessary graph transforming, and avoid introducing any additional incorrect dependencies.
Generally if you use LEFT/RIGHT then the input rows are going to be available wherever they are used. However if they are passed into a function, and that function uses them inside a definition marked as global then that is invalid (since by definition global expressions don't have any context).
Similarly if you use syntax <dataset>.<field>, its validity and meaning depends on whether <dataset> is active. The scope transformer ensures that all references to fields are legal, and adds a "new" attribute to any no_selects where it is necessary.
This transform simplifies the expression tree. Its aim is to simplify scalar expressions, and dataset expressions that are valid whether or not the nodes are shared. Some examples are:
1 + 2 => 3 and any other operation on scalar constants.
IF(true, x, y) => x
COUNT(<empty-dataset>) => 0
IF (a = b, 'c', 'd') = 'd' => IF(a=b, false, true) => a != b
Simplifying sorts, projects filters on empty datasets
Most of the optimizations are fairly standard, but a few have been added to cover more esoteric examples which have occurred in queries over the years.
This transform also supports the option to percolate constants through the graph. E.g., if a project assigns the value 3 to a field, it can substitute the value 3 wherever that field is used in subsequent activities. This can often lead to further opportunities for constant folding (and removing fields in the implicit project).
This transformer is used to simplify, combine and reorder dataset expressions. The transformer takes care to count the number of times each expression is used to ensure that none of the transformations cause duplication. E.g., swapping a filter with a sort is a good idea, but if there are two filters of the same sort and they are both swapped you will now be duplicating the sort.
ECL tends to be written as general purpose definitions which can then be combined. This can lead to potential inefficiencies - e.g., one definition may summarise some data in 20 different ways, this is then used by another definition which only uses a subset of those results. The implicit project transformer tracks the data flow at each point through the expression graph, and removes any fields that are not required.
This often works in combination with the other optimizations. For instance the constant percolation can remove the need for fields, and removing fields can sometimes allow a left outer join to be converted to a project.
is this the correct term? Should it be a query? This should really be independent of this document...)
The code generator ultimately creates workunits. A workunit completely describes a generated query. It consists of two parts. There is an xml component - this contains the workflow information, the various execution graphs, and information about options. It also describes which inputs can be supplied to the query and what results are generated. The other part is the generated shared object compiled from the generated C++. This contains functions and classes that are used by the engines to execute the queries. Often the xml is compressed and stored as a resource within the shared object -so the shared object contains a complete workunit.
The actions in a workunit are divided up into individual workflow items. Details of when each workflow item is executed, what its dependencies are stored in the <Workflow> section of the xml. The generated code also contains a class definition, with a method perform() which is used to execute the actions associated with a particular workflow item. (The class instances are created by calling the exported createProcess() factory function).
The generated code for an individual workflow item will typically call back into the engine at some point to execute a graph.
The activity graphs are stored in the xml. The graph contains details of which activities are required, how those activities link together, what dependencies there are between the activities. For each activity it might contain the following information:
A unique id.
The "kind" of the activity (from enum ThorActivityKind in eclhelper.hpp)
The ECL that created the activity.
Name of the original definition
Location (e.g., file, line number) of the original ECL.
Information about the record size, number of rows, sort order etc.
Hints which control options for a particular activity (e.g,, the number of threads to use while sorting).
Record counts and stats once the job has executed.
Each activity in a graph also has a corresponding helper class instance in the generated code. (The name of the class is cAc followed by the activity number, and the exported factory method is fAc followed by the activity number.) These classes implement the interfaces defined in eclhelper.hpp.
The engine uses the information from the xml to produce a graph of activities that need to be executed. It has a general purpose implementation of each activity kind, and it uses the class instance to tailor that general activity to the specific use e.g., what is the filter condition, what fields are set up, what is the sort order?
The workunit xml contains details of what inputs can be supplied when that workunit is run. These correspond to STORED definitions in the ECL. The result xml also contains the schema for the results that the workunit will generate.
Once an instance of the workunit has been run, the values of the results may be written back into dali's copy of the workunit so they can be retrieved and displayed.
Compile time is an issue - especially for small on-demand queries. To help reduce compile times (and dependencies with the rest of the system) the number of header files included by the generated code is kept to a minimum. In particular references to jlib, boost and icu are kept within the implementation of the runtime functions, and are not included in the public dependencies.
Thread-safe.
It should be possible to use the members of an activity helper from multiple threads without issue. The helpers may contain some context dependent state, so different instances of the helpers are needed for concurrent use from different contexts (e.g., expansions of a graph.)
Concise.
The code should be broadly readable, but the variable names etc. are chosen to generate compact code.
Functional.
Generally the generated code assigns to a variable once, and doesn't modify it afterwards. Some assignments may be conditional, but once the variable is evaluated it isn't updated. (There are of course a few exceptions - e.g., dataset iterators)
First a few pointers to help understand the code within eclcc:
It makes extensive use of link counting. You need understand that concept to get very far.
If something is done more than once then that is generally split into a helper function.
The helper functions aren't generally added to the corresponding interface (e.g., IHqlExpression) because the interface would become bloated. Instead they are added as global functions. The big disadvantage of this approach is they can be hard to find. Even better would be for them to be rationalised and organised into namespaces.
The code is generally thread-safe unless there would be a significant performance implication. In generally all the code used by the parser for creating expressions is thread safe. Expression graph transforms are thread-safe, and can execute in parallel if a constant (NUM_PARALLEL_TRANSFORMS) is increased. The data structures used to represent the generated code are NOT thread-safe.
Much of the code generation is structured fairly procedurally, with classes used to process the stages within it.
There is a giant "God" class HqlCppTranslator - which could really do with refactoring.
The eclcc parser uses the standard tools bison and flex to process the ECL and convert it to a
: expression graph. There are a couple of idiosyncrasies with the way it is implemented.
Macros with fully qualified scope.
Slightly unusually macros are defined in the same way that other definitions are - in particular to can have references to macros in other modules. This means that there are references to macros within the grammar file (instead of being purely handled by a pre-processor). It also means the lexer keeps an active stack of macros being processed.
Attributes on operators.
Many of the operators have optional attributes (e.g., KEEP, INNER, LOCAL, ...). If these were all reserved words it would remove a significant number of keywords from use as symbols, and could also mean that when a new attribute was added it broke existing code. To avoid this the lexer looks ahead in the parser tables (by following the potential reductions) to see if the token really could come next. If it can't then it isn't reserved as a symbol.
As the workunit is created the code generator builds up the generated code and the xml for the workunit. Most of the xml generation is encapsulated within the IWorkUnit interface. The xml for the graphs is created in an IPropertyTree, and added to the workunit as a block.
The C++ generation is ultimately controlled by some template files (thortpl.cpp). The templates are plain text and contain references to allow named sections of code to be expanded at particular points.
The code generator builds up some structures in memory for each of those named sections. Once the generation is complete some peephole optimization is applied to the code. This structure is walked to expand each named section of code as required.
The BuildCtx class provides a cursor into that generated C++. It will either be created for a given named section, or more typically from another BuildCtx. It has methods for adding the different types of statements. Some are simple (e.g., addExpr()), whilst some create a compound statement (e.g., addFilter). The compound statements change the active selector so any new statements are added within that compound statement.
As well as building up a tree of expressions, this data structure also maintains a tree of associations. For instance when a value is evaluated and assigned to a temporary variable, the logical value is associated with that temporary. If the same expression is required later, the association is matched, and the temporary value is used instead of recalculating it. The associations are also used to track the active datasets, classes generated for row-meta information, activity classes etc. etc.
Each activity in an expression graph will have an associated class generated in the C++. Each different activity kind expects a helper that implements a particular IHThorArg interface. E.g., a sort activity of kind TAKsort requires a helper that implements IHThorSortArg. The associated factory function is used to create instances of the helper class.
The generated class might take one of two forms:
A parameterised version of a library class. These are generated for simple helpers that don't have many variations (e.g., CLibrarySplitArg for TAKsplit), or for special cases that occur very frequently (CLibraryWorkUnitReadArg for internal results).
A class derived from a skeleton implementation of that helper (typically CThorXYZ implementing interface IHThorXYZ). The base class has default implementations of some of the functions, and any exceptions are implemented in the derived class.
This is a class that is used by the engines to encapsulate all the information about a single row -e.g., the format that each activity generates. It is an implementation of the IOutputMeta interface. It includes functions to
The same expression nodes are used for representing expressions in the generated C++ as the original ECL expression graph. It is important to keep track of whether an expression represents untranslated ECL, or the "translated" C++. For instance ECL has 1 based indexes, while C++ is zero based. If you processed the expression x[1] it might get translated to x[0] in C++. Translating it again would incorrectly refer to x[-1].
There are two key classes used while building the C++ for an ECL expression:
CHqlBoundExpr.
This represents a value that has been converted to C++. Depending on the type, one or more of the fields will be filled in.
CHqlBoundTarget.
This represents the target of an assignment -C++ variable(s) that are going to be assigned the result of evaluating an expression. It is almost always passed as a const parameter to a function because the target is well-defined and the function needs to update that target.
A C++ expression is sometimes converted back to an ECL pseudo-expression by calling getTranslatedExpr(). This creates an expression node of kind no_translated to indicate the child expression has already been converted.
The generation code for expressions has a hierarchy of calls. Each function is there to allow optimal code to be generated - e.g., not creating a temporary variable if none are required. A typical flow might be:
buildExpr(ctx, expr, bound).
Evaluate the ecl expression "expr" and save the C++ representation in the class bound. This might then call through to...
buildTempExpr(ctx, expr, bound);
Create a temporary variable, and evaluate expr and assign it to that temporary variable.... Which then calls.
buildExprAssign(ctx, target, expr);
evaluate the expression, and ensure it is assigned to the C++ target "target".
The default implementation might be to call buildExpr....
An operator must either be implemented in buildExpr() (calling a function called doBuildExprXXX) or in buildExprAssign() (calling a function called doBuildAssignXXX). Some operators are implemented in both places if there are different implementations that would be more efficient in each context.
Similarly there are several different assignment functions:
buildAssign(ctx, <ecl-target>, <ecl-value>);
buildExprAssign(ctx, <c++-target>, <ecl-value>);
assign(ctx, <C++target>, <c++source>)
The different varieties are there depending on whether the source value or targets have already been translated. (The names could be rationalised!)
Most dataset operations are only implemented as activities (e.g., PARSE, DEDUP). If these are used within a transform/filter then eclcc will generate a call to a child query. An activity helper for the appropriate operation will then be generated.
However a subset of the dataset operations can also be evaluated inline without calling a child query. Some examples are filters, projects, and simple aggregation. It removes the overhead of the child query call in the simple cases, and often generates more concise code.
When datasets are evaluated inline there is a similar hierarchy of function calls:
buildDatasetAssign(ctx, target, expr);
Evaluate the dataset expression, and assign it to the target (a builder interface). This may then call....
buildIterate(ctx, expr)
Iterate through each of the rows in the dataset expression in turn. Which may then call...
buildDataset(ctx, expr, target, format)
Build the entire dataset, and return it as a single value.
Some of the operations (e.g., aggregating a filtered dataset) can be done more efficiently by summing and filtering an iterator, than forcing the filtered dataset to be evaluated first.
The interface IHqlCppDatasetCursor allows the code generator to iterate through a dataset, or select a particular element from a dataset. It is used to hide the different representation of datasets, e.g.,
Blocked - the rows are in a contiguous block of memory appended one after another.
Array - the dataset is represented by an array of pointers to the individual rows.
Link counted - similar to array, but each element is also link counted.
Nested. Sometimes the cursor may iterate through multiple levels of child datasets.
Generally rows that are serialized (e.g., on disk) are in blocked format, and they are stored as link counted rows in memory.
The IReferenceSelector interface and the classes in hqltcppc[2] provide an interface for getting and setting values within a row of a dataset. They hide the details of the layout - e.g., csv/xml/raw data, and the details of exactly how each type is represented in the row.
The current implementation of keys in HPCC uses a format which uses a separate 8 byte integer field which was historically used to store the file position in the original file. Other complications are that the integer fields are stored big-endian, and signed integer values are biased.
This introduces some complication in the way indexes are handled. You will often find that the logical index definition is replaced with a physical index definition, followed by a project to convert it to the logical view. A similar process occurs for disk files to support VIRTUAL(FILEPOSITION) etc.
As mentioned at the start of this document, one of the main challenges with eclcc is converting the declarative ECL code into imperative C++ code. The direction we are heading in is to allow the engines to support more lazy-evaluation so possibly in this instance to evaluate it the first time it is used (although that may potentially be much less efficient). This will allow the code generator to relax some of its current assumptions.
There are several example queries which are already producing pathological behaviour from eclcc, causing it to generate C++ functions which are many thousands of lines long.
Currently the grammar for the parser is too specialised. In particular the separate productions for expression, datasets, actions cause problems - e.g., it is impossible to properly allow sets of datasets to be treated in the same way as other sets.
The semantic checking (and probably semantic interpretation) is done too early. Really the parser should build up a syntax tree, and then disambiguate it and perform the semantic checks on the syntax tree.
The function calls should probably be expanded later than they are. I have tried in the past and hit problems, but I can't remember all the details. Some are related to the semantic checking.
`,223)]))}const f=t(r,[["render",s]]);export{u as __pageData,f as default};
diff --git a/assets/devdoc_CodeReviews.md.BB4WmPQp.js b/assets/devdoc_CodeReviews.md.BB4WmPQp.js
new file mode 100644
index 00000000000..7f3ac71a5e8
--- /dev/null
+++ b/assets/devdoc_CodeReviews.md.BB4WmPQp.js
@@ -0,0 +1 @@
+import{_ as t,c as i,a3 as o,o as a}from"./chunks/framework.DkhCEVKm.js";const m=JSON.parse('{"title":"Code Review Guidelines","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/CodeReviews.md","filePath":"devdoc/CodeReviews.md","lastUpdated":1731340314000}'),s={name:"devdoc/CodeReviews.md"};function r(n,e,d,l,h,c){return a(),i("div",null,e[0]||(e[0]=[o('
The Code Submissions document is aimed at developers that are submitting PRs. This document describes some of the goals and expectations for code reviewers.
Catch architectural or design problems. These should have been caught earlier, but better later than never...
Catch bugs early (incorrect behaviour, inefficiencies, security issues)
Ensure the code is readable and maintainable. This includes following the project coding standards (see Style Guide).
A opportunity for training/passing on information. For example providing information about how the current system works, functionality that is already available or suggestions of other approaches the developer may not have thought of.
It is NOT a goal to change the submission until it matches how the reviewer would have coded it.
Code reviewers should be explicit and clearly describe the problem. This should include what change is expected if not obvious. Don’t assume the contributor has same understanding/view as reviewer.
If a comment is not clear the contributor should ask for clarification. ...rather than wasting time trying to second-guess the reviewer.
Contributors should feel free to push back if they consider comments are too picky. The reviewer can either agree, or provide reasons why they consider it to be an issue.
The reviewer should not extend the scope of the original change. If the change could be extended, or only partially solves the issue, a new JIRA should be created for the extra work. If the change will introduce regressions, or fundamentally fails to solve the problem then this does not apply!
Clearly indicate if a review is incomplete. Sometimes a significant design problem means the rest of the code has not been reviewed in detail. Other times an initial review has picked up a set of issues, but the reviewer needs to go back and check other aspects in detail. If this is the case it should be explicitly noted.
Repeated issues. The reviewer is free to comment on every instance of a repeated issue, but a simple annotation should alert contributor to address appropriately eg: [Please address all instances of this issue]
Contributers should provide feedback to the reviewer. The contributor should respond to a comment if it isn't obvious where/how they have been addressed (but no need to acknowledge typo/indentation/etc)
Only the reviewer should mark issues as resolved using the Github resolve conversation button.
Code reviews should be a priority. Both reviewers and contributors should respond in a timely manner - don't leave it for days. It destroys the flow of thought and conversation.
Check all review comments have been addressed. If they have not been addressed you are guaranteed another review/submit cycle. In particular watch out for collapsed conversations. If there are large numbers of comments GitHub will collapse them, which can make comments easy to miss.
Sometimes PRs need to be restarted. If there are large number of comments > 100, it can be hard to track all the comments and GitHub can become unresponsive. It may be better to close the PR and open a new one.
Submit any changes as extra commits. This makes it clear to the reviewer what has changed, and avoids them having to re-review everything. Please do not squash them until the reviewers approve the PR. The few exceptions to this are if the PR is only a couple of lines, or the PR is completely rewritten in response to the review.
Reviewers use GitHub's features Making use of the "viewed" button can make it easier to track what has changed - or quickly remove trivial changes from view. Ignoring whitespace can often simplify comparisons - especially when code has been refactored or extra conditions or try/catch bocks have been introduced.
All code reviews don't need to be equally strict. The "strictness" of the review should reflect the importance and location of the change. Some examples:
If it is closely associated with an existing file, then the indentation, style should match the surrounding code - a mixture of styles makes it much harder to read. If it is in a new, independent source file or project this is less of an issue.
If the code is in a core library then efficiency and edge cases will be more important.
If it is a core part of the system then security is key. If it is a developer only tool then edge cases are less significant.
Reviews of draft pull requests are likely to concentrate on the overall approach, rather than the details. They are likely to be more informal (e.g. not always using comments tags).
Does it introduce any memory leaks? E.g. Correct use of linking? Are exceptions released?
Thread safety
critical sections or atomic variables if accessed by more than one thread
race conditions
deadlock
authorization. Should it be checked, does it fail by default?
Any potential for overflow or DOS? Are all user inputs validated and all lengths protected?
Are all secrets stored and passed securely?
Comments explaining why for any code that is complex or counter-intuitive.
Backward compatibility. Could this possibly cause problems if data produced with this change is used in earlier/later versions? Could there be problems if it was used in a mixed-version environment?
When reading comments in a review it can sometimes be hard to know why the reviewer made a comment, or what response is expected. If there is any doubt the contributor should ask. However to make it clearer we are aiming to always add a tag to the front of each review comment. The tag will give an indication of why the comment is being made, its severity and what kind of response is expected. Here is a provisional table of tags:
Tag
What
Why
Expected response
design:
An architectural or design issue
The reviewer considers the PR has a significant problem which will affect its functionality or future extensibility
reviewer/developer redesign expected before any further changes
scope:
The scope of the PR does not match the Jira
If the scope of the fix is too large it can be hard to review, and take much longer to resolve all the issues before the PR is accepted.
Discussion. Split the PR into multiple simpler PRs.
function:
Incorrect/unexpected functionality implemented
The function doesn't match the description in the jira, or doesn't solve the original problem
developer expected to address issue (or discuss)
security:
Something in the code introduces a security problem
The reviewer has spotted potential security issues e.g. injection attacks
developer expected to discuss the issue (and then address)
bug:
A coding issue that will cause incorrect behaviour
Likely to cause confusion, invalid results or crashes.
developer expected to address issue
efficiency:
The code works, but may have scaling or other efficiency issues.
Inefficiency can cause problem in some key functions and areas
developer addressing the problem (or discuss)
discuss:
Reviewer has thought of a potential problem, but not sure if it applies
Reviewer has a concern it may be an issue, and wants to check the developer has thought about and addressed the issue
Discussion - either in the PR or offline.
style:
Reviewer points out non-conforming code style
Makes the code hard to read
Developer to fix
indent:
A fairly obvious indentation issue
Makes the code hard to read
Developer to fix.
format:
Any other unusual formatting
Makes the code hard to read
Developer to fix.
typo:
Minor typing error
Makes something (code/message/comment) harder to read
Developer to fix.
minor:
A minor issue that could be improved.
Education (the suggestion is better for a particular reason), or something simple to clean up at the same time as other changes
Developer recommended to fix, but unlikely to stop a merge
picky:
A very minor issue that could be improved, but is barely worth commenting on
Education, or something to clean up at the same time as other changes
Developer discretion to fix, wouldn't stop a merge
future:
An additional feature or functionality that fits in but should be done as a separate PR.
Ensure that missing functionality is tracked, but PRs are not held up by additional requirements.
Contributor to create Jira (unless trivial) and number noted in response.
question:
Review has a question that they are not sure of the answer to
Reviewer would like clarification to help understand the code or design. The answer may lead to further comments.
An answer to the question.
note:
Reviewer wants to pass on some information to the contributor which they may not know
Passing on knowledge/background
contributor should consider the note, but no change expected/required
personal:
Reviewer has an observation based on personal experience
Reviewer has comments that would improve the code, but not part of the style guide or required. E.g. patterns for guard conditions
Reflect on the suggestion, but no change expected.
documentation:
This change may have an impact on documentation
Make sure changes can be used
Contributor to create Jira describing the impact created, and number noted in response.
The comments should always be constructive. The reviewer should have a reason for each of them, and be able to articulate the reason in the comment or when asked. "I wouldn't have done it like that" is not a good enough on its own!
Similarly there is a difference in opinion within the team on some style issues - e.g. standard libraries or jlib, inline or out of line functions, nested or non-nested classes. Reviews should try and avoid commenting on these unless there is a clear reason why they are significant (functionality, efficiency, compile time) and if so spell it out. Code reviewers should discuss any style issues that they consider should be universally adopted that are not in the style guide.
',23)]))}const p=t(s,[["render",r]]);export{m as __pageData,p as default};
diff --git a/assets/devdoc_CodeReviews.md.BB4WmPQp.lean.js b/assets/devdoc_CodeReviews.md.BB4WmPQp.lean.js
new file mode 100644
index 00000000000..7f3ac71a5e8
--- /dev/null
+++ b/assets/devdoc_CodeReviews.md.BB4WmPQp.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as i,a3 as o,o as a}from"./chunks/framework.DkhCEVKm.js";const m=JSON.parse('{"title":"Code Review Guidelines","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/CodeReviews.md","filePath":"devdoc/CodeReviews.md","lastUpdated":1731340314000}'),s={name:"devdoc/CodeReviews.md"};function r(n,e,d,l,h,c){return a(),i("div",null,e[0]||(e[0]=[o('
The Code Submissions document is aimed at developers that are submitting PRs. This document describes some of the goals and expectations for code reviewers.
Catch architectural or design problems. These should have been caught earlier, but better later than never...
Catch bugs early (incorrect behaviour, inefficiencies, security issues)
Ensure the code is readable and maintainable. This includes following the project coding standards (see Style Guide).
A opportunity for training/passing on information. For example providing information about how the current system works, functionality that is already available or suggestions of other approaches the developer may not have thought of.
It is NOT a goal to change the submission until it matches how the reviewer would have coded it.
Code reviewers should be explicit and clearly describe the problem. This should include what change is expected if not obvious. Don’t assume the contributor has same understanding/view as reviewer.
If a comment is not clear the contributor should ask for clarification. ...rather than wasting time trying to second-guess the reviewer.
Contributors should feel free to push back if they consider comments are too picky. The reviewer can either agree, or provide reasons why they consider it to be an issue.
The reviewer should not extend the scope of the original change. If the change could be extended, or only partially solves the issue, a new JIRA should be created for the extra work. If the change will introduce regressions, or fundamentally fails to solve the problem then this does not apply!
Clearly indicate if a review is incomplete. Sometimes a significant design problem means the rest of the code has not been reviewed in detail. Other times an initial review has picked up a set of issues, but the reviewer needs to go back and check other aspects in detail. If this is the case it should be explicitly noted.
Repeated issues. The reviewer is free to comment on every instance of a repeated issue, but a simple annotation should alert contributor to address appropriately eg: [Please address all instances of this issue]
Contributers should provide feedback to the reviewer. The contributor should respond to a comment if it isn't obvious where/how they have been addressed (but no need to acknowledge typo/indentation/etc)
Only the reviewer should mark issues as resolved using the Github resolve conversation button.
Code reviews should be a priority. Both reviewers and contributors should respond in a timely manner - don't leave it for days. It destroys the flow of thought and conversation.
Check all review comments have been addressed. If they have not been addressed you are guaranteed another review/submit cycle. In particular watch out for collapsed conversations. If there are large numbers of comments GitHub will collapse them, which can make comments easy to miss.
Sometimes PRs need to be restarted. If there are large number of comments > 100, it can be hard to track all the comments and GitHub can become unresponsive. It may be better to close the PR and open a new one.
Submit any changes as extra commits. This makes it clear to the reviewer what has changed, and avoids them having to re-review everything. Please do not squash them until the reviewers approve the PR. The few exceptions to this are if the PR is only a couple of lines, or the PR is completely rewritten in response to the review.
Reviewers use GitHub's features Making use of the "viewed" button can make it easier to track what has changed - or quickly remove trivial changes from view. Ignoring whitespace can often simplify comparisons - especially when code has been refactored or extra conditions or try/catch bocks have been introduced.
All code reviews don't need to be equally strict. The "strictness" of the review should reflect the importance and location of the change. Some examples:
If it is closely associated with an existing file, then the indentation, style should match the surrounding code - a mixture of styles makes it much harder to read. If it is in a new, independent source file or project this is less of an issue.
If the code is in a core library then efficiency and edge cases will be more important.
If it is a core part of the system then security is key. If it is a developer only tool then edge cases are less significant.
Reviews of draft pull requests are likely to concentrate on the overall approach, rather than the details. They are likely to be more informal (e.g. not always using comments tags).
Does it introduce any memory leaks? E.g. Correct use of linking? Are exceptions released?
Thread safety
critical sections or atomic variables if accessed by more than one thread
race conditions
deadlock
authorization. Should it be checked, does it fail by default?
Any potential for overflow or DOS? Are all user inputs validated and all lengths protected?
Are all secrets stored and passed securely?
Comments explaining why for any code that is complex or counter-intuitive.
Backward compatibility. Could this possibly cause problems if data produced with this change is used in earlier/later versions? Could there be problems if it was used in a mixed-version environment?
When reading comments in a review it can sometimes be hard to know why the reviewer made a comment, or what response is expected. If there is any doubt the contributor should ask. However to make it clearer we are aiming to always add a tag to the front of each review comment. The tag will give an indication of why the comment is being made, its severity and what kind of response is expected. Here is a provisional table of tags:
Tag
What
Why
Expected response
design:
An architectural or design issue
The reviewer considers the PR has a significant problem which will affect its functionality or future extensibility
reviewer/developer redesign expected before any further changes
scope:
The scope of the PR does not match the Jira
If the scope of the fix is too large it can be hard to review, and take much longer to resolve all the issues before the PR is accepted.
Discussion. Split the PR into multiple simpler PRs.
function:
Incorrect/unexpected functionality implemented
The function doesn't match the description in the jira, or doesn't solve the original problem
developer expected to address issue (or discuss)
security:
Something in the code introduces a security problem
The reviewer has spotted potential security issues e.g. injection attacks
developer expected to discuss the issue (and then address)
bug:
A coding issue that will cause incorrect behaviour
Likely to cause confusion, invalid results or crashes.
developer expected to address issue
efficiency:
The code works, but may have scaling or other efficiency issues.
Inefficiency can cause problem in some key functions and areas
developer addressing the problem (or discuss)
discuss:
Reviewer has thought of a potential problem, but not sure if it applies
Reviewer has a concern it may be an issue, and wants to check the developer has thought about and addressed the issue
Discussion - either in the PR or offline.
style:
Reviewer points out non-conforming code style
Makes the code hard to read
Developer to fix
indent:
A fairly obvious indentation issue
Makes the code hard to read
Developer to fix.
format:
Any other unusual formatting
Makes the code hard to read
Developer to fix.
typo:
Minor typing error
Makes something (code/message/comment) harder to read
Developer to fix.
minor:
A minor issue that could be improved.
Education (the suggestion is better for a particular reason), or something simple to clean up at the same time as other changes
Developer recommended to fix, but unlikely to stop a merge
picky:
A very minor issue that could be improved, but is barely worth commenting on
Education, or something to clean up at the same time as other changes
Developer discretion to fix, wouldn't stop a merge
future:
An additional feature or functionality that fits in but should be done as a separate PR.
Ensure that missing functionality is tracked, but PRs are not held up by additional requirements.
Contributor to create Jira (unless trivial) and number noted in response.
question:
Review has a question that they are not sure of the answer to
Reviewer would like clarification to help understand the code or design. The answer may lead to further comments.
An answer to the question.
note:
Reviewer wants to pass on some information to the contributor which they may not know
Passing on knowledge/background
contributor should consider the note, but no change expected/required
personal:
Reviewer has an observation based on personal experience
Reviewer has comments that would improve the code, but not part of the style guide or required. E.g. patterns for guard conditions
Reflect on the suggestion, but no change expected.
documentation:
This change may have an impact on documentation
Make sure changes can be used
Contributor to create Jira describing the impact created, and number noted in response.
The comments should always be constructive. The reviewer should have a reason for each of them, and be able to articulate the reason in the comment or when asked. "I wouldn't have done it like that" is not a good enough on its own!
Similarly there is a difference in opinion within the team on some style issues - e.g. standard libraries or jlib, inline or out of line functions, nested or non-nested classes. Reviews should try and avoid commenting on these unless there is a clear reason why they are significant (functionality, efficiency, compile time) and if so spell it out. Code reviewers should discuss any style issues that they consider should be universally adopted that are not in the style guide.
',23)]))}const p=t(s,[["render",r]]);export{m as __pageData,p as default};
diff --git a/assets/devdoc_CodeSubmissions.md.DoLTBfxy.js b/assets/devdoc_CodeSubmissions.md.DoLTBfxy.js
new file mode 100644
index 00000000000..d6cbfdc68f3
--- /dev/null
+++ b/assets/devdoc_CodeSubmissions.md.DoLTBfxy.js
@@ -0,0 +1 @@
+import{_ as t,c as i,a3 as o,o as s}from"./chunks/framework.DkhCEVKm.js";const m=JSON.parse('{"title":"Code Submission Guidelines","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/CodeSubmissions.md","filePath":"devdoc/CodeSubmissions.md","lastUpdated":1731340314000}'),r={name:"devdoc/CodeSubmissions.md"};function a(n,e,h,l,d,c){return s(),i("div",null,e[0]||(e[0]=[o('
We welcome submissions to the platform especially in the form of pull requests into the HPCC-Systems github repository. The following describes some of processes for merging PRs.
There are a few things that should be considered when creating a PR to increase the likelihood that they can be accepted quickly.
Write a good commit message The format should be HPCC-XXXXX (where XXXXX is the bug number) followed by a description of the issue. The text should make sense in a change log by itself - without reference to the jira or the contents of the PR. We should aim to increase the information that is included as part of the commit message - not rely on on the jira.
Ensure the reviewer has enough information to review the change. The code reviewer only has the JIRA and the PR to go on. The JIRA (or associated documentation) should contain enough details to review the PR - e.g. the purpose, main aim, why the change was made etc.. If the scope of the jira has changed then the jira should be updated to reflect that. If the submission requires changes to the documentation then the JIRA should contain all the details needed to document it, and the PR should either contain the documentation changes, or a documentation JIRA should be created.
Fill in the checklist The check boxes are there to remind you to consider different aspects of the PR. Not all of them apply to every submission, but if you tick a box and have not really thought about the item then prepare to be embarrassed!
Prefer small submissions It isn't always possible, but several smaller PRs are much easier to review than one large change. If your submission includes semi-automatic/mechanical changes (e.g. renaming large numbers of function calls, or adding an extra parameter) please keep it as a separate commit. This makes it much easier to review the PR - since the reviewer will be looking for different errors in the different types of changes.
Check for silly mistakes Review your own code in github, after creating the PR to check for silly mistakes. It doesn't take long, and often catches trivial issues. It may avoid the need for a cycle of code-review/fixes. It may be helpful to add some notes to specific changes e.g. "this change is mainly or solely refactoring method A into method B and C. ". Some common examples of trivial issues to look for include:
Inconsistent indentation, or using tabs rather than spaces to indent.
Lines of tracing left in.
Lines of code commented out that should be deleted.
TBD reminders of work that need implementing or removing.
Unrelated files that have been accidentally modified.
Accidental changes to submodule versions.
Typos in error messages, tracing or comments, or in the commit message.
Incomplete edits when copy and pasting code.
Check subtractions are the right way around, and potential for overflow.
New files with the wrong copyright date
Check the target branch (see below)
Request one or more reviews. For relatively simple changes a single reviewer is normally enough.
All pull requests should be reviewed by someone who is not the author before merging. Complex changes, changes that require input from multiple experts, or that have implications throughout the system should be reviewed by multiple reviewers. This should include someone who is responsible for merging changes for that part of the system. (Unless it is a simple change written by someone with review rights.)
Contributors should use the github reviewers section on the PR to request reviews. After a contributor has pushed a set of changes in response to a review, they should refresh the github review status, so the users are notified it is ready for re-review. When the review is complete, a person with responsibility for merging changes to that part of the system should be added as a reviewer (or refreshed), with a comment that it is ready to merge.
Reviewers should check for PRs that are ready for their review via github's webpage (filter "review-requested:<reviewer-id>") or via the github CLI (e.g. gh pr status). Contributors should similarly ensure they stay up to date with any comments on requests for change on their submissions.
The Version support document contains details of the different versions that are supported, and which version should be targetted for different kinds of changes. Occasionally earlier branches will be chosen, (e.g. security fixes to even older versions) but they should always be carefully discussed (and documented).
Changes will always be upmerged into the next point release for all the more recent major and minor versions (and master).
',12)]))}const f=t(r,[["render",a]]);export{m as __pageData,f as default};
diff --git a/assets/devdoc_CodeSubmissions.md.DoLTBfxy.lean.js b/assets/devdoc_CodeSubmissions.md.DoLTBfxy.lean.js
new file mode 100644
index 00000000000..d6cbfdc68f3
--- /dev/null
+++ b/assets/devdoc_CodeSubmissions.md.DoLTBfxy.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as i,a3 as o,o as s}from"./chunks/framework.DkhCEVKm.js";const m=JSON.parse('{"title":"Code Submission Guidelines","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/CodeSubmissions.md","filePath":"devdoc/CodeSubmissions.md","lastUpdated":1731340314000}'),r={name:"devdoc/CodeSubmissions.md"};function a(n,e,h,l,d,c){return s(),i("div",null,e[0]||(e[0]=[o('
We welcome submissions to the platform especially in the form of pull requests into the HPCC-Systems github repository. The following describes some of processes for merging PRs.
There are a few things that should be considered when creating a PR to increase the likelihood that they can be accepted quickly.
Write a good commit message The format should be HPCC-XXXXX (where XXXXX is the bug number) followed by a description of the issue. The text should make sense in a change log by itself - without reference to the jira or the contents of the PR. We should aim to increase the information that is included as part of the commit message - not rely on on the jira.
Ensure the reviewer has enough information to review the change. The code reviewer only has the JIRA and the PR to go on. The JIRA (or associated documentation) should contain enough details to review the PR - e.g. the purpose, main aim, why the change was made etc.. If the scope of the jira has changed then the jira should be updated to reflect that. If the submission requires changes to the documentation then the JIRA should contain all the details needed to document it, and the PR should either contain the documentation changes, or a documentation JIRA should be created.
Fill in the checklist The check boxes are there to remind you to consider different aspects of the PR. Not all of them apply to every submission, but if you tick a box and have not really thought about the item then prepare to be embarrassed!
Prefer small submissions It isn't always possible, but several smaller PRs are much easier to review than one large change. If your submission includes semi-automatic/mechanical changes (e.g. renaming large numbers of function calls, or adding an extra parameter) please keep it as a separate commit. This makes it much easier to review the PR - since the reviewer will be looking for different errors in the different types of changes.
Check for silly mistakes Review your own code in github, after creating the PR to check for silly mistakes. It doesn't take long, and often catches trivial issues. It may avoid the need for a cycle of code-review/fixes. It may be helpful to add some notes to specific changes e.g. "this change is mainly or solely refactoring method A into method B and C. ". Some common examples of trivial issues to look for include:
Inconsistent indentation, or using tabs rather than spaces to indent.
Lines of tracing left in.
Lines of code commented out that should be deleted.
TBD reminders of work that need implementing or removing.
Unrelated files that have been accidentally modified.
Accidental changes to submodule versions.
Typos in error messages, tracing or comments, or in the commit message.
Incomplete edits when copy and pasting code.
Check subtractions are the right way around, and potential for overflow.
New files with the wrong copyright date
Check the target branch (see below)
Request one or more reviews. For relatively simple changes a single reviewer is normally enough.
All pull requests should be reviewed by someone who is not the author before merging. Complex changes, changes that require input from multiple experts, or that have implications throughout the system should be reviewed by multiple reviewers. This should include someone who is responsible for merging changes for that part of the system. (Unless it is a simple change written by someone with review rights.)
Contributors should use the github reviewers section on the PR to request reviews. After a contributor has pushed a set of changes in response to a review, they should refresh the github review status, so the users are notified it is ready for re-review. When the review is complete, a person with responsibility for merging changes to that part of the system should be added as a reviewer (or refreshed), with a comment that it is ready to merge.
Reviewers should check for PRs that are ready for their review via github's webpage (filter "review-requested:<reviewer-id>") or via the github CLI (e.g. gh pr status). Contributors should similarly ensure they stay up to date with any comments on requests for change on their submissions.
The Version support document contains details of the different versions that are supported, and which version should be targetted for different kinds of changes. Occasionally earlier branches will be chosen, (e.g. security fixes to even older versions) but they should always be carefully discussed (and documented).
Changes will always be upmerged into the next point release for all the more recent major and minor versions (and master).
',12)]))}const f=t(r,[["render",a]]);export{m as __pageData,f as default};
diff --git a/assets/devdoc_DevDocs.md.DbnKa3rJ.js b/assets/devdoc_DevDocs.md.DbnKa3rJ.js
new file mode 100644
index 00000000000..18a1bd08827
--- /dev/null
+++ b/assets/devdoc_DevDocs.md.DbnKa3rJ.js
@@ -0,0 +1,8 @@
+import{_ as t,c as a,a3 as o,o as i}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Working with developer documentation","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/DevDocs.md","filePath":"devdoc/DevDocs.md","lastUpdated":1731340314000}'),n={name:"devdoc/DevDocs.md"};function s(d,e,l,r,c,h){return i(),a("div",null,e[0]||(e[0]=[o(`
Documents can be located anywhere in the repository folder structure. If it makes sense to have documentation "close" to specific components, then it can be located in the same folder as the component. For example, any developer documentation for specific plugins can be located in those folders. If this isn't appropriate then the documentation can be located in the devdoc or subfolders of devdoc.
WARNING
There is an exclusion list in the devdoc/.vitepress/config.js file that prevents certain folders from being included in the documentation. If you add a new document to a folder that is excluded, then it will not be included in the documentation. If you need to add a new document to an excluded folder, then you will need to update the exclusion list in the devdoc/.vitepress/config.js file.
Documentation is written in Markdown. This is a simple format that is easy to read and write. It is also easy to convert to other formats, such as HTML, PDF, and Word. Markdown is supported by many editors, including Visual Studio Code, and is supported by VitePress.
TIP
VitePress extends Markdown with some additional features, such as custom containers, it is recommended that you refer to the VitePress documentation for more details.
To assist with the writing of documentation, VitePress can be used to render the documentation locally. This allows you to see how the documentation will look when it is published. To start the local development server you need to type the following commands in the root HPCC-Platform folder:
sh
npm install
+npm run docs-dev
This will start a local development server and display the URL that you can use to view the documentation. The default URL is http://localhost:5173/HPCC-Platform, but it may be different on your machine. The server will automatically reload when you make changes to the documentation.
WARNING
The first time you start the VitePress server it will take a while to complete. This is because it is locating all the markdown files in the repository and creating the html pages. Once it has completed this step once, it will be much faster to start the server again.
To add a new document, you need to add a new markdown file to the repository. The file should be named appropriately and have the .md file extension. Once the file exists, you can view it by navigating to the appropriate URL. For example, if you add a new file called MyNewDocument.md to the devdoc folder, then you can view it by navigating to http://localhost:5173/HPCC-Platform/devdoc/MyNewDocument.html.
To add a new document to the sidebar, you need to add an entry to the devdoc/.vitepress/config.js file. The entry should be added to the sidebar section. For example, to add a new document called MyNewDocument.md to the devdoc folder, you would add the following entry to the sidebar section:
The conent of the main landing page is located in index.md in the root folder. Its structure uses the VitePress "home" layout.
`,21)]))}const m=t(n,[["render",s]]);export{u as __pageData,m as default};
diff --git a/assets/devdoc_DevDocs.md.DbnKa3rJ.lean.js b/assets/devdoc_DevDocs.md.DbnKa3rJ.lean.js
new file mode 100644
index 00000000000..18a1bd08827
--- /dev/null
+++ b/assets/devdoc_DevDocs.md.DbnKa3rJ.lean.js
@@ -0,0 +1,8 @@
+import{_ as t,c as a,a3 as o,o as i}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Working with developer documentation","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/DevDocs.md","filePath":"devdoc/DevDocs.md","lastUpdated":1731340314000}'),n={name:"devdoc/DevDocs.md"};function s(d,e,l,r,c,h){return i(),a("div",null,e[0]||(e[0]=[o(`
Documents can be located anywhere in the repository folder structure. If it makes sense to have documentation "close" to specific components, then it can be located in the same folder as the component. For example, any developer documentation for specific plugins can be located in those folders. If this isn't appropriate then the documentation can be located in the devdoc or subfolders of devdoc.
WARNING
There is an exclusion list in the devdoc/.vitepress/config.js file that prevents certain folders from being included in the documentation. If you add a new document to a folder that is excluded, then it will not be included in the documentation. If you need to add a new document to an excluded folder, then you will need to update the exclusion list in the devdoc/.vitepress/config.js file.
Documentation is written in Markdown. This is a simple format that is easy to read and write. It is also easy to convert to other formats, such as HTML, PDF, and Word. Markdown is supported by many editors, including Visual Studio Code, and is supported by VitePress.
TIP
VitePress extends Markdown with some additional features, such as custom containers, it is recommended that you refer to the VitePress documentation for more details.
To assist with the writing of documentation, VitePress can be used to render the documentation locally. This allows you to see how the documentation will look when it is published. To start the local development server you need to type the following commands in the root HPCC-Platform folder:
sh
npm install
+npm run docs-dev
This will start a local development server and display the URL that you can use to view the documentation. The default URL is http://localhost:5173/HPCC-Platform, but it may be different on your machine. The server will automatically reload when you make changes to the documentation.
WARNING
The first time you start the VitePress server it will take a while to complete. This is because it is locating all the markdown files in the repository and creating the html pages. Once it has completed this step once, it will be much faster to start the server again.
To add a new document, you need to add a new markdown file to the repository. The file should be named appropriately and have the .md file extension. Once the file exists, you can view it by navigating to the appropriate URL. For example, if you add a new file called MyNewDocument.md to the devdoc folder, then you can view it by navigating to http://localhost:5173/HPCC-Platform/devdoc/MyNewDocument.html.
To add a new document to the sidebar, you need to add an entry to the devdoc/.vitepress/config.js file. The entry should be added to the sidebar section. For example, to add a new document called MyNewDocument.md to the devdoc folder, you would add the following entry to the sidebar section:
The conent of the main landing page is located in index.md in the root folder. Its structure uses the VitePress "home" layout.
`,21)]))}const m=t(n,[["render",s]]);export{u as __pageData,m as default};
diff --git a/assets/devdoc_Development.md.Cz70vUo4.js b/assets/devdoc_Development.md.Cz70vUo4.js
new file mode 100644
index 00000000000..db4c016916b
--- /dev/null
+++ b/assets/devdoc_Development.md.Cz70vUo4.js
@@ -0,0 +1,3 @@
+import{_ as s,c as t,a3 as i,o as a}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Development Guide","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/Development.md","filePath":"devdoc/Development.md","lastUpdated":1731340314000}'),n={name:"devdoc/Development.md"};function o(h,e,l,r,p,d){return a(),t("div",null,e[0]||(e[0]=[i(`
The HPCC Platform sources are hosted on GitHub. You can download a snapshot of any branch using the download button there, or you can set up a git clone of the repository. If you are planning to contribute changes to the system, see the CONTRIBUTORS document for information about how to set up a GitHub fork of the project through which pull-requests can be made.
The HPCC platform requires a number of third party tools and libraries in order to build. The HPCC Wiki contains the details of the dependencies that are required for different distributions.
For building any documentation, the following are also required:
The HPCC system is built using the cross-platform build tool cmake, which is available for Windows, virtually all flavors of Linux, FreeBSD, and other platforms. You should install cmake version 2.8.3 or later before building the sources.
On some distros you will need to build cmake from sources if the version of cmake in the standard repositories for that distro is not modern enough. It is good practice in cmake to separate the build directory where objects and executables are made from the source directory, and the HPCC cmake scripts will enforce this.
To build the sources, create a directory where the built files should be located, and from that directory, run:
bash
cmake <source directory>
Depending on your operating system and the compilers installed on it, this will create a makefile, Visual Studio .sln file, or other build script for building the system. If cmake was configured to create a makefile, then you can build simply by typing:
bash
make
If a Visual Studio solution file was created, you can load it simply by typing the name: hpccsystems-platform.sln
This will load the solution in Visual Studio where you can build in the usual way.
To make an installation package on a supported linux system, use the command:
bash
make package
This will first do a make to ensure everything is up to date, then will create the appropriate package for your operating system, Currently supported package formats are rpm (for RedHat/Centos) and .deb (for Debian and Ubuntu). If the operating system is not one of the above, or is not recognized, make package will create a tarball.
The package installation does not start the service on the machine, so if you want to give it a go or test it (see below), make sure to start the service manually and wait until all services are up (mainly wait for EclWatch to come up on port 8010).
Some components have their own unit-tests. Once you have compiled (no need to start the services), you can already run them. Supposing you build a Debug version, from the build directory you can run:
The ECLCC compiler tests rely on two distinct runs: a known good one and your test build. For normal development, you can safely assume that the OSS/master branch in github is good. For overnight testing, golden directories need to be maintained according to the test infrastructure. There are Bash (Linux) and Batch (Windows) scripts to run the regressions:
The basic idea behind this tests is to compare the output files (logs and XML files) between runs. The log files should change slightly (the comparison should be good enough to filter most irrelevant differences), but the XML files should be identical if nothing has changed. You should only see differences in the XML where you have changed in the code, or new tests were added as part of your development.
On Linux, there are two steps:
Step 1: Check-out OSS/master, compile and run the regressions to populate the 'golden' directory:
bash
./regress.sh -t golden -e buildDir/Debug/bin/eclcc
This will run the regressions in parallel, using as many CPUs as you have, and using your just-compiled ECLCC, assuming you compiled for Debug version.
Step 2: Make your changes (or check-out your branch), compile and run again, this time output to a new directory and compare to the 'golden' repo.:
bash
./regress.sh -t my_branch -c golden -e buildDir/Debug/bin/eclcc
This will run the regressions in the same way, output to 'my_branch' dir and compare it to the golden version, highlighting the differences.
NOTE: If you changed the headers that the compiled binaries will use, you must re-install the package (or provide -i option to the script to the new headers).
Step 3: Step 2 only listed the differences, now you need to see what they are. For that, re-run the regressing script omitting the compiler, since the only thing we'll do is to compare verbosely.:
bash
./regress.sh -t my_branch -c golden
This will show you all differences, using the same ignore filters as before, between your two branches. Once you're happy with the differences, commit and issue a pull-request.
On linux systems, the makefile generated by cmake will build a specific version (debug or release) of the system depending on the options selected when cmake is first run in that directory. The default is to build a release system. In order to build a debug system instead, use command:
bash
cmake -DCMAKE_BUILD_TYPE=Debug <source directory>
You can then run make or make package in the usual way to build the system.
On a Windows system, cmake always generates s solution file with both debug and release target platforms in it, so you can select which one to build within Visual Studio.
`,56)]))}const g=s(n,[["render",o]]);export{u as __pageData,g as default};
diff --git a/assets/devdoc_Development.md.Cz70vUo4.lean.js b/assets/devdoc_Development.md.Cz70vUo4.lean.js
new file mode 100644
index 00000000000..db4c016916b
--- /dev/null
+++ b/assets/devdoc_Development.md.Cz70vUo4.lean.js
@@ -0,0 +1,3 @@
+import{_ as s,c as t,a3 as i,o as a}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Development Guide","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/Development.md","filePath":"devdoc/Development.md","lastUpdated":1731340314000}'),n={name:"devdoc/Development.md"};function o(h,e,l,r,p,d){return a(),t("div",null,e[0]||(e[0]=[i(`
The HPCC Platform sources are hosted on GitHub. You can download a snapshot of any branch using the download button there, or you can set up a git clone of the repository. If you are planning to contribute changes to the system, see the CONTRIBUTORS document for information about how to set up a GitHub fork of the project through which pull-requests can be made.
The HPCC platform requires a number of third party tools and libraries in order to build. The HPCC Wiki contains the details of the dependencies that are required for different distributions.
For building any documentation, the following are also required:
The HPCC system is built using the cross-platform build tool cmake, which is available for Windows, virtually all flavors of Linux, FreeBSD, and other platforms. You should install cmake version 2.8.3 or later before building the sources.
On some distros you will need to build cmake from sources if the version of cmake in the standard repositories for that distro is not modern enough. It is good practice in cmake to separate the build directory where objects and executables are made from the source directory, and the HPCC cmake scripts will enforce this.
To build the sources, create a directory where the built files should be located, and from that directory, run:
bash
cmake <source directory>
Depending on your operating system and the compilers installed on it, this will create a makefile, Visual Studio .sln file, or other build script for building the system. If cmake was configured to create a makefile, then you can build simply by typing:
bash
make
If a Visual Studio solution file was created, you can load it simply by typing the name: hpccsystems-platform.sln
This will load the solution in Visual Studio where you can build in the usual way.
To make an installation package on a supported linux system, use the command:
bash
make package
This will first do a make to ensure everything is up to date, then will create the appropriate package for your operating system, Currently supported package formats are rpm (for RedHat/Centos) and .deb (for Debian and Ubuntu). If the operating system is not one of the above, or is not recognized, make package will create a tarball.
The package installation does not start the service on the machine, so if you want to give it a go or test it (see below), make sure to start the service manually and wait until all services are up (mainly wait for EclWatch to come up on port 8010).
Some components have their own unit-tests. Once you have compiled (no need to start the services), you can already run them. Supposing you build a Debug version, from the build directory you can run:
The ECLCC compiler tests rely on two distinct runs: a known good one and your test build. For normal development, you can safely assume that the OSS/master branch in github is good. For overnight testing, golden directories need to be maintained according to the test infrastructure. There are Bash (Linux) and Batch (Windows) scripts to run the regressions:
The basic idea behind this tests is to compare the output files (logs and XML files) between runs. The log files should change slightly (the comparison should be good enough to filter most irrelevant differences), but the XML files should be identical if nothing has changed. You should only see differences in the XML where you have changed in the code, or new tests were added as part of your development.
On Linux, there are two steps:
Step 1: Check-out OSS/master, compile and run the regressions to populate the 'golden' directory:
bash
./regress.sh -t golden -e buildDir/Debug/bin/eclcc
This will run the regressions in parallel, using as many CPUs as you have, and using your just-compiled ECLCC, assuming you compiled for Debug version.
Step 2: Make your changes (or check-out your branch), compile and run again, this time output to a new directory and compare to the 'golden' repo.:
bash
./regress.sh -t my_branch -c golden -e buildDir/Debug/bin/eclcc
This will run the regressions in the same way, output to 'my_branch' dir and compare it to the golden version, highlighting the differences.
NOTE: If you changed the headers that the compiled binaries will use, you must re-install the package (or provide -i option to the script to the new headers).
Step 3: Step 2 only listed the differences, now you need to see what they are. For that, re-run the regressing script omitting the compiler, since the only thing we'll do is to compare verbosely.:
bash
./regress.sh -t my_branch -c golden
This will show you all differences, using the same ignore filters as before, between your two branches. Once you're happy with the differences, commit and issue a pull-request.
On linux systems, the makefile generated by cmake will build a specific version (debug or release) of the system depending on the options selected when cmake is first run in that directory. The default is to build a release system. In order to build a debug system instead, use command:
bash
cmake -DCMAKE_BUILD_TYPE=Debug <source directory>
You can then run make or make package in the usual way to build the system.
On a Windows system, cmake always generates s solution file with both debug and release target platforms in it, so you can select which one to build within Visual Studio.
`,56)]))}const g=s(n,[["render",o]]);export{u as __pageData,g as default};
diff --git a/assets/devdoc_GitAuthenticate.md.DWDVKYp3.js b/assets/devdoc_GitAuthenticate.md.DWDVKYp3.js
new file mode 100644
index 00000000000..cfaef864e5f
--- /dev/null
+++ b/assets/devdoc_GitAuthenticate.md.DWDVKYp3.js
@@ -0,0 +1,12 @@
+import{_ as a,c as t,a3 as s,o as n}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"HPCC git support","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/GitAuthenticate.md","filePath":"devdoc/GitAuthenticate.md","lastUpdated":1731340314000}'),i={name:"devdoc/GitAuthenticate.md"};function o(p,e,r,c,l,h){return n(),t("div",null,e[0]||(e[0]=[s(`
Version 8.4 of the HPCC platform allows package files to define dependencies between git repositories and also allows you to compile directly from a git repository.
E.g.
ecl run hthor --main demo.main@ghalliday/gch-ecldemo-d#version1 --server=...
There are no futher requirements if the repositories are public, but private repositories have the additional complication of supplying authentication information. Git provides various methods for providing the credentials...
This is used when the github reference is of the form ssh://github.com. The sshkey can be protected with a passcode, and there are various options to avoid having to enter the passcode each time.
It is preferrable to use the https:// protocol instead of ssh:// for links in package-lock.json files. If ssh:// is used it requires any machine that processes the dependency to have access to a registered ssh key.
github authentication
Download the GitHub command line tool (https://github.com/cli/cli). You can then use it to authenticate all git access with
All of the options above are likely to involve some user interaction - passphrases for ssh keys, web interaction with github authentication, and initial entry for cached access tokens. This is problematic for eclccserver - which cannot support user interaction, and it is preferrable not to pass credentials around.
The solution is to use a personal access token securely stored as a secret. (This would generally be associated with a special service account.) This avoids the need to pass credentials and allows the keys to be rotated.
The following describes the support in the different versions:
b) add a secret to the values.yaml file, with a key that matches the username:
secrets:
+ git:
+ ghalliday: my-git-secret
note: this cannot currently use a vault - probably need to rethink that. (Possibly extract from secret and supply as an optional environment variable to be picked up by the bash script.)
c) add a secret to Kubernetes containing the personal access token:
`,41)]))}const g=a(i,[["render",o]]);export{u as __pageData,g as default};
diff --git a/assets/devdoc_GitAuthenticate.md.DWDVKYp3.lean.js b/assets/devdoc_GitAuthenticate.md.DWDVKYp3.lean.js
new file mode 100644
index 00000000000..cfaef864e5f
--- /dev/null
+++ b/assets/devdoc_GitAuthenticate.md.DWDVKYp3.lean.js
@@ -0,0 +1,12 @@
+import{_ as a,c as t,a3 as s,o as n}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"HPCC git support","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/GitAuthenticate.md","filePath":"devdoc/GitAuthenticate.md","lastUpdated":1731340314000}'),i={name:"devdoc/GitAuthenticate.md"};function o(p,e,r,c,l,h){return n(),t("div",null,e[0]||(e[0]=[s(`
Version 8.4 of the HPCC platform allows package files to define dependencies between git repositories and also allows you to compile directly from a git repository.
E.g.
ecl run hthor --main demo.main@ghalliday/gch-ecldemo-d#version1 --server=...
There are no futher requirements if the repositories are public, but private repositories have the additional complication of supplying authentication information. Git provides various methods for providing the credentials...
This is used when the github reference is of the form ssh://github.com. The sshkey can be protected with a passcode, and there are various options to avoid having to enter the passcode each time.
It is preferrable to use the https:// protocol instead of ssh:// for links in package-lock.json files. If ssh:// is used it requires any machine that processes the dependency to have access to a registered ssh key.
github authentication
Download the GitHub command line tool (https://github.com/cli/cli). You can then use it to authenticate all git access with
All of the options above are likely to involve some user interaction - passphrases for ssh keys, web interaction with github authentication, and initial entry for cached access tokens. This is problematic for eclccserver - which cannot support user interaction, and it is preferrable not to pass credentials around.
The solution is to use a personal access token securely stored as a secret. (This would generally be associated with a special service account.) This avoids the need to pass credentials and allows the keys to be rotated.
The following describes the support in the different versions:
b) add a secret to the values.yaml file, with a key that matches the username:
secrets:
+ git:
+ ghalliday: my-git-secret
note: this cannot currently use a vault - probably need to rethink that. (Possibly extract from secret and supply as an optional environment variable to be picked up by the bash script.)
c) add a secret to Kubernetes containing the personal access token:
This document covers the main steps taken by the LDAP Security Manager during initialization. It is important to note that the LDAP Security Manager uses the LDAP protocol to access an Active Directory, AD. The AD is the store for users, groups, permissions, resources, and more. The term LDAP is generally overused to refer to both.
Each service and/or component using the LDAP security manager gets its own instance of the security manager. This includes a unique connection pool (see below). All operations described apply to each LDAP instance.
The LDAP Security Manager supports using multiple ADs. The FQDN or IP address of each AD host is read from configuration data and stored internally. The source is a comma separated list stored in the ldapAddress config value. Each entry is added to a pool of ADs.
Note that all ADs are expected to use the same credentials and have the same configuration
AD credentials consist of a username and password. The LDAP security manager users these to perform all operations on behalf of users and components in the cluster. There are three potential sources for credentials.
A Kubernetes secret. If the ldapAdminSecretKey config value is set, but ldapAdminVaultId is not (see 2) then the AD credentials are retrieved as Kubernetes secrets.
If both ldapAdminSecretKey and ldapAdminVaultId config values are present, the AD credentials are retrieved from the vault.
Hardcoded values from the systemCommonName and systemPassword config values stored in the environment.xml file.
As stated above, when multiple ADs are in use, the configuration of each must be the same. This includes credentials.
During initialization, the security manager begins incrementing through the set of defined ADs until it is able to connect and retrieve information from an AD. Once retrieved, the information is used for all ADs (see statement above about all ADs being the same). The accessed AD is marked as the current AD and no other ADs are accessed during initialization.
The retrieved information is used to verify the AD type so the security manager adjusts for variations between types. Additionally, defined DNs may be adjusted to match AD type requirements.
The manager handles connections to an AD in order to perform required operations. It is possible that values such as permissions and resources may be cached to improve performance.
The LDAP security manager maintains a pool of LDAP connections. The pool is limited in size to maxConnections from the configuration. The connection pool starts empty. As connections are created, each is added to the pool until the max allowed is reached.
The following process is used when an LDAP connection is needed.
First, the connection pool is searched for a free connection. If found and valid, the connection is returned. A connection is considered free if no one is using it and valid if the AD can be accessed. If no valid free connections are found, a new uninitialized connection is created.
For a new connection, an attempt is made to connect to each AD starting with the current. See Handling AD Hosts below for how ADs are cycled when a connection fails. For each AD, as it cycles through, connection attempts are retried with a short delay between each. If unable to connect, the AD host is marked rejected and the next is attempted. Once a new connection has been established, if the max number of connections has not been reached yet, the connection is added to the pool.
It is important to note that if the pool has reached its max size, new connections will continue to be made, but are not saved in the pool. This allows the pool to maintain a steady working state, but allow for higher demand. Connections not saved to the pool are deleted once no longer in use.
The manager keeps a list of AD hosts and the index of the current host. The current host is used for all AD operations until there is a failure. At that time the manager marks the host as "rejected" and moves to the next host using a round-robin scheme.
',28)]))}const p=t(i,[["render",r]]);export{m as __pageData,p as default};
diff --git a/assets/devdoc_LDAPSecurityManager.md.Cty5AMLN.lean.js b/assets/devdoc_LDAPSecurityManager.md.Cty5AMLN.lean.js
new file mode 100644
index 00000000000..77a97406441
--- /dev/null
+++ b/assets/devdoc_LDAPSecurityManager.md.Cty5AMLN.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as a,a3 as n,o}from"./chunks/framework.DkhCEVKm.js";const m=JSON.parse('{"title":"LDAP Security Manager Init","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/LDAPSecurityManager.md","filePath":"devdoc/LDAPSecurityManager.md","lastUpdated":1731340314000}'),i={name:"devdoc/LDAPSecurityManager.md"};function r(s,e,c,d,h,l){return o(),a("div",null,e[0]||(e[0]=[n('
This document covers the main steps taken by the LDAP Security Manager during initialization. It is important to note that the LDAP Security Manager uses the LDAP protocol to access an Active Directory, AD. The AD is the store for users, groups, permissions, resources, and more. The term LDAP is generally overused to refer to both.
Each service and/or component using the LDAP security manager gets its own instance of the security manager. This includes a unique connection pool (see below). All operations described apply to each LDAP instance.
The LDAP Security Manager supports using multiple ADs. The FQDN or IP address of each AD host is read from configuration data and stored internally. The source is a comma separated list stored in the ldapAddress config value. Each entry is added to a pool of ADs.
Note that all ADs are expected to use the same credentials and have the same configuration
AD credentials consist of a username and password. The LDAP security manager users these to perform all operations on behalf of users and components in the cluster. There are three potential sources for credentials.
A Kubernetes secret. If the ldapAdminSecretKey config value is set, but ldapAdminVaultId is not (see 2) then the AD credentials are retrieved as Kubernetes secrets.
If both ldapAdminSecretKey and ldapAdminVaultId config values are present, the AD credentials are retrieved from the vault.
Hardcoded values from the systemCommonName and systemPassword config values stored in the environment.xml file.
As stated above, when multiple ADs are in use, the configuration of each must be the same. This includes credentials.
During initialization, the security manager begins incrementing through the set of defined ADs until it is able to connect and retrieve information from an AD. Once retrieved, the information is used for all ADs (see statement above about all ADs being the same). The accessed AD is marked as the current AD and no other ADs are accessed during initialization.
The retrieved information is used to verify the AD type so the security manager adjusts for variations between types. Additionally, defined DNs may be adjusted to match AD type requirements.
The manager handles connections to an AD in order to perform required operations. It is possible that values such as permissions and resources may be cached to improve performance.
The LDAP security manager maintains a pool of LDAP connections. The pool is limited in size to maxConnections from the configuration. The connection pool starts empty. As connections are created, each is added to the pool until the max allowed is reached.
The following process is used when an LDAP connection is needed.
First, the connection pool is searched for a free connection. If found and valid, the connection is returned. A connection is considered free if no one is using it and valid if the AD can be accessed. If no valid free connections are found, a new uninitialized connection is created.
For a new connection, an attempt is made to connect to each AD starting with the current. See Handling AD Hosts below for how ADs are cycled when a connection fails. For each AD, as it cycles through, connection attempts are retried with a short delay between each. If unable to connect, the AD host is marked rejected and the next is attempted. Once a new connection has been established, if the max number of connections has not been reached yet, the connection is added to the pool.
It is important to note that if the pool has reached its max size, new connections will continue to be made, but are not saved in the pool. This allows the pool to maintain a steady working state, but allow for higher demand. Connections not saved to the pool are deleted once no longer in use.
The manager keeps a list of AD hosts and the index of the current host. The current host is used for all AD operations until there is a failure. At that time the manager marks the host as "rejected" and moves to the next host using a round-robin scheme.
',28)]))}const p=t(i,[["render",r]]);export{m as __pageData,p as default};
diff --git a/assets/devdoc_MemoryManager.md.lMxAbcSH.js b/assets/devdoc_MemoryManager.md.lMxAbcSH.js
new file mode 100644
index 00000000000..5a70e95fd80
--- /dev/null
+++ b/assets/devdoc_MemoryManager.md.lMxAbcSH.js
@@ -0,0 +1 @@
+import{_ as a,c as t,a3 as o,o as i}from"./chunks/framework.DkhCEVKm.js";const p=JSON.parse('{"title":"The Roxie Memory Manager","description":"","frontmatter":{"title":"The Roxie Memory Manager"},"headers":[],"relativePath":"devdoc/MemoryManager.md","filePath":"devdoc/MemoryManager.md","lastUpdated":1731340314000}'),r={name:"devdoc/MemoryManager.md"};function s(n,e,l,h,c,d){return i(),t("div",null,e[0]||(e[0]=[o('
This memory manager started life as the memory manager which was only used for the Roxie engine. It had several original design goals:
Support link counted rows. (When the last reference is released the row is freed.)
Be as fast as possible on allocate and deallocate of small rows.
Allow rows serialized from slaves to be used directly without being cloned first.
Allow the memory used by a single query, or by all queries combined, to be limited, with graceful recovery.
Isolate roxie queries from one another, so that one query can't bring down all the rest by allocating too much memory.
Guarantee all the memory used by a query is freed when the query finishes, reducing the possibility of memory leaks.
Predictable behaviour with no pathogenic cases.
(Note that efficient usage of memory does not appear on that list - the expectation when the memory manager was first designed was that Roxie queries would use minimal amounts of memory and speed was more important. Some subsequent changes e.g., Packed heaps, and configurable bucket sizes help mitigate that.)
The basic design is to reserve (but not commit) a single large block of memory in the virtual address space. This memory is subdivided into "pages". (These are not the same as the os virtual memory pages. The memory manager pages are currently defined as 1Mb in size.)
The system uses a bitmap to indicate whether each page from the global memory has been allocated. All active IRowManager instances allocate pages from the same global memory space. To help reduce fragmentation allocations for single pages are fulfilled from one end of the address space, while allocations for multiple pages are fulfilled from the other.
This provides the primary interface for allocating memory. The size of a requested allocation is rounded up to the next "bucket" size, and the allocation is then satisfied by the heap associated with that bucket size. Different engines can specify different bucket sizes - an optional list is provided to setTotalMemoryLimit. Roxie tends to use fewer buckets to help reduce the number of active heaps. Thor uses larger numbers since it is more important to minimize the memory wasted.
Roxie uses a separate instance of IRowManager for each query. This provides the mechanism for limiting how much memory a query uses. Thor uses a single instance of an IRowManager for each slave/master.
Memory is allocated from a set of "heaps - where each heap allocates blocks of memory of a single size. The heap exclusively owns a set of heaplet (each 1 page in size), which are held in a doubly linked list, and sub allocates memory from those heaplets.
Information about each heaplet is stored in the base of the page (using a class with virtual functions) and the address of an allocated row is masked to determine which heap object it belongs to, and how it should be linked/released etc. Any pointer not in the allocated virtual address (e.g., constant data) can be linked/released with no effect.
Each heaplet contains a high water mark of the address within the page that has already been allocated (freeBase), and a lockless singly-linked list of rows which have been released (r_block). Releasing a row is non-blocking and does not involve any spin locks or critical sections. However, this means that empty pages need to be returned to the global memory pool at another time. (This is done in releaseEmptyPages()).
When the last row in a page is released a flag (possibleEmptyPages) is set in its associated heap. * This is checked before trying to free pages from a particular heap, avoiding waiting on a lock and traversing a candidate list.
Any page which might contain some spare memory is added to a lockless spare memory linked list. * Items are popped from this list when a heap fails to allocate memory from the current heaplet. Each item is checked in turn if it has space before allocating a new heaplet. * The list is also walked when checking to see which pages can be returned to the global memory. The doubly linked heaplet list allows efficient freeing.
Each allocation has a link count and an allocator id associated with it. The allocator id represents the type of the row, and is used to determine what destructor needs to be called when the row is destroyed. (The count for a row also contains a flag in the top bit to indicate if it is fully constructed, and therefore valid for the destructor to be called.)
A specialized heap is used to manage all allocations that require more than one page of memory. These allocations are not held on a free list when they are released, but each is returned directly to the global memory pool. Allocations in the huge heap can be expanded and shrunk using the resizeRow() functions - see below.
By default a fixed size heaps rounds the requested allocation size up to the next bucket size. A packed heap changes this behaviour and it is rounded up to the next 4 byte boundary instead. This reduces the amount of memory wasted for each row, but potentially increases the number of distinct heaps.
By default all fixed size heaps of the same size are shared. This reduces the memory consumption, but if the heap is used by multiple threads it can cause significant contention. If a unique heap is specified then it will not be shared with any other requests. Unique heaps store information about the type of each row in the heaplet header, rather than per row - which reduces the allocation overhead for each row. (Note to self: Is there ever any advantage having a heap that is unique but not packed??)
Blocked is an option on createFixedRowHeap() to allocate multiple rows from the heaplet, and then return the additional rows on subsequent calls. It is likely to reduce the average number of atomic operations required for each row being allocated, but the heap that is returned can only be used from a single thread because it is not thread safe.
By default the heaplets use a lock free singly linked list to keep track of rows that have been freed. This requires an atomic operation for each allocation and for each free. The scanning allocator uses an alternative approach. When a row is freed the row header is marked, and a row is allocated by scanning through the heaplet for rows that have been marked as free. Scanning uses atomic store and get, rather than more expensive synchronized atomic operations, so is generally faster than the linked list - provided a free row is found fairly quickly.
The scanning heaps have better best-case performance, but worse worse-case performance (if large numbers of rows need to be scanned before a free row is found). The best-case tends to be true if only one thread/activity is accessing a particular heap, and the worse-case if multiple activities are accessing a heap, particularly if the rows are being buffered. It is the default for thor which tends to have few active allocators, but not for roxie, which tends to have large numbers of allocators.
This is another varation on the scanning allocator, which further reduces the number of atomic operations. Usually when a row is allocated the link count on the heaplet is increased, and when it is freed the link count is decremented. This option delays decrementing the link count when the row is released, by marking the row with a different free flag. If it is subsequently reallocated there is no need to increment the link count. The downside is that it is more expensive to check whether a heaplet is completely empty (since you can no longer rely on the heaplet linkcount alone).
Thor has additional requirements to roxie. In roxie, if a query exceeds its memory requirements then it is terminated. Thor needs to be able to spill rows and other memory to disk and continue. This is achieved by allowing any process that stores buffered rows to register a callback with the memory manager. When more memory is required these callbacks are called to free up memory, and allow the job to continue.
Each callback can specify a priority - lower priority callbacks are called first since they are assumed to have a lower cost associated with spilling. When more memory is required the callbacks are called in priority order until one of them succeeds. The can also be passed a flag to indicate it is critical to force them to free up as much memory as possible.
A callback cannot allocate any memory from the memory manager. If it does it is likely to deadlock.
You cannot allocate memory while holding a lock if that lock is also required by a callback.
Again this will cause deadlock. If it proves impossible you can use a try-lock primitive in the callback, but it means you won't be able to spill those rows.
If the heaps are fragmented it may be more efficient to repack the heaps than spill to disk.
If you're resizing a potentially big block of memory use the resize function with the callback.
Some of the memory allocations cover more than one "page" - e.g., arrays used to store blocks of rows. (These are called huge pages internally, not to be confused with operating system support for huge pages...) When one of these memory blocks needs to be expanded you need to be careful:
Allocating a new page, copying, updating the pointer (within a cs) and then freeing is safe. Unfortunately it may involve copying a large chunk of memory. It may also fail if there isn't memory for the new and old block, even if the existing block could have been expanded into an adjacent block.
You can't lock, call a resize routine and update the pointer because the resize routine may need to allocate a new memory block- that may trigger a callback, which could in turn deadlock trying to gain the lock. (The callback may be from another thread...)
Therefore the memory manager contains a call which allows you to resize a block, but with a callback which is used to atomically update the pointer so it always remains thread safe.
Occasionally you have processes which read a large number of rows and then filter them so only a few are still held in memory. Rows tend to be allocated in sequence through the heap pages, which can mean those few remaining rows are scattered over many pages. If they could all be moved to a single page it would free up a significant amount of memory.
The memory manager contains a function to pack a set of rows into a smaller number of pages: IRowManager->compactRows().
This works by iterating through each of the rows in a list. If the row belongs to a heap that could be compacted, and isn't part of a full heaplet, then the row is moved. Since subsequent rows tend to be allocated from the same heaplet this has the effect of compacting the rows.
Much of the time Thor doesn't uses full memory available to it. If you are running multiple Thor processes on the same machine you may want to configure the system so that each Thor has a private block of memory, but there is also a shared block of memory which can be used by whichever process needs it.
The ILargeMemCallback provides a mechanism to dynamically allocate more memory to a process as it requires it. This could potentially be done in stages rather than all or nothing.
(Currently unused as far as I know... the main problem is that borrowing memory needs to be coordinated.)
When OS processes use a large amount of memory, mapping virtual addresses to physical addresses can begin to take a significant proportion of the execution time. This is especially true once the TLB is not large enough to store all the mappings. Huge pages can significantly help with this problem by reducing the number of TLB entries needed to cover the virtual address space. The memory manager supports huge pages in two different ways:
Huge pages can be preallocated (e.g., with hugeadm) for exclusive use as huge pages. If huge pages are enabled for a particular engine, and sufficient huge pages are available to supply the memory for the memory manager, then they will be used.
Linux kernels from 2.6.38 onward have support for transparent huge pages. These do not need to be preallocated, instead the operating system tries to use them behind the scenes. HPCC version 5.2 and following takes advantage of this feature to significantly speed memory access up when large amounts of memory are used by each process.
Preallocated huge pages tend to be more efficient, but they have the disadvantage that the operating system currently does not reuse unused huge pages for other purposes e.g., disk cache.
There is also a memory manager option to not return the memory to the operating system when it is no longer required. This has the advantage of not clearing the memory whenever it is required again, but the same disadvantage as preallocated huge pages that the unused memory cannot be used for disk cache. We recommend this option is selected when preallocated huge pages are in use - until the kernel allows them to be reused.
Changes in 6.x allow Thor to run multiple channels within the same process. This allows data that is constant for all channels to be shared between all slave channels - a prime example is the rhs of a lookup join. For the queries to run efficiently the memory manager needs to ensure that each slave channel has the same amount of memory - especially when memory is being used that is shared between them.
createGlobalRowManager() allows a single global row manager to be created which also provides slave row managers for the different channels via the querySlaveRowManager(unsigned slave) method.
',61)]))}const u=a(r,[["render",s]]);export{p as __pageData,u as default};
diff --git a/assets/devdoc_MemoryManager.md.lMxAbcSH.lean.js b/assets/devdoc_MemoryManager.md.lMxAbcSH.lean.js
new file mode 100644
index 00000000000..5a70e95fd80
--- /dev/null
+++ b/assets/devdoc_MemoryManager.md.lMxAbcSH.lean.js
@@ -0,0 +1 @@
+import{_ as a,c as t,a3 as o,o as i}from"./chunks/framework.DkhCEVKm.js";const p=JSON.parse('{"title":"The Roxie Memory Manager","description":"","frontmatter":{"title":"The Roxie Memory Manager"},"headers":[],"relativePath":"devdoc/MemoryManager.md","filePath":"devdoc/MemoryManager.md","lastUpdated":1731340314000}'),r={name:"devdoc/MemoryManager.md"};function s(n,e,l,h,c,d){return i(),t("div",null,e[0]||(e[0]=[o('
This memory manager started life as the memory manager which was only used for the Roxie engine. It had several original design goals:
Support link counted rows. (When the last reference is released the row is freed.)
Be as fast as possible on allocate and deallocate of small rows.
Allow rows serialized from slaves to be used directly without being cloned first.
Allow the memory used by a single query, or by all queries combined, to be limited, with graceful recovery.
Isolate roxie queries from one another, so that one query can't bring down all the rest by allocating too much memory.
Guarantee all the memory used by a query is freed when the query finishes, reducing the possibility of memory leaks.
Predictable behaviour with no pathogenic cases.
(Note that efficient usage of memory does not appear on that list - the expectation when the memory manager was first designed was that Roxie queries would use minimal amounts of memory and speed was more important. Some subsequent changes e.g., Packed heaps, and configurable bucket sizes help mitigate that.)
The basic design is to reserve (but not commit) a single large block of memory in the virtual address space. This memory is subdivided into "pages". (These are not the same as the os virtual memory pages. The memory manager pages are currently defined as 1Mb in size.)
The system uses a bitmap to indicate whether each page from the global memory has been allocated. All active IRowManager instances allocate pages from the same global memory space. To help reduce fragmentation allocations for single pages are fulfilled from one end of the address space, while allocations for multiple pages are fulfilled from the other.
This provides the primary interface for allocating memory. The size of a requested allocation is rounded up to the next "bucket" size, and the allocation is then satisfied by the heap associated with that bucket size. Different engines can specify different bucket sizes - an optional list is provided to setTotalMemoryLimit. Roxie tends to use fewer buckets to help reduce the number of active heaps. Thor uses larger numbers since it is more important to minimize the memory wasted.
Roxie uses a separate instance of IRowManager for each query. This provides the mechanism for limiting how much memory a query uses. Thor uses a single instance of an IRowManager for each slave/master.
Memory is allocated from a set of "heaps - where each heap allocates blocks of memory of a single size. The heap exclusively owns a set of heaplet (each 1 page in size), which are held in a doubly linked list, and sub allocates memory from those heaplets.
Information about each heaplet is stored in the base of the page (using a class with virtual functions) and the address of an allocated row is masked to determine which heap object it belongs to, and how it should be linked/released etc. Any pointer not in the allocated virtual address (e.g., constant data) can be linked/released with no effect.
Each heaplet contains a high water mark of the address within the page that has already been allocated (freeBase), and a lockless singly-linked list of rows which have been released (r_block). Releasing a row is non-blocking and does not involve any spin locks or critical sections. However, this means that empty pages need to be returned to the global memory pool at another time. (This is done in releaseEmptyPages()).
When the last row in a page is released a flag (possibleEmptyPages) is set in its associated heap. * This is checked before trying to free pages from a particular heap, avoiding waiting on a lock and traversing a candidate list.
Any page which might contain some spare memory is added to a lockless spare memory linked list. * Items are popped from this list when a heap fails to allocate memory from the current heaplet. Each item is checked in turn if it has space before allocating a new heaplet. * The list is also walked when checking to see which pages can be returned to the global memory. The doubly linked heaplet list allows efficient freeing.
Each allocation has a link count and an allocator id associated with it. The allocator id represents the type of the row, and is used to determine what destructor needs to be called when the row is destroyed. (The count for a row also contains a flag in the top bit to indicate if it is fully constructed, and therefore valid for the destructor to be called.)
A specialized heap is used to manage all allocations that require more than one page of memory. These allocations are not held on a free list when they are released, but each is returned directly to the global memory pool. Allocations in the huge heap can be expanded and shrunk using the resizeRow() functions - see below.
By default a fixed size heaps rounds the requested allocation size up to the next bucket size. A packed heap changes this behaviour and it is rounded up to the next 4 byte boundary instead. This reduces the amount of memory wasted for each row, but potentially increases the number of distinct heaps.
By default all fixed size heaps of the same size are shared. This reduces the memory consumption, but if the heap is used by multiple threads it can cause significant contention. If a unique heap is specified then it will not be shared with any other requests. Unique heaps store information about the type of each row in the heaplet header, rather than per row - which reduces the allocation overhead for each row. (Note to self: Is there ever any advantage having a heap that is unique but not packed??)
Blocked is an option on createFixedRowHeap() to allocate multiple rows from the heaplet, and then return the additional rows on subsequent calls. It is likely to reduce the average number of atomic operations required for each row being allocated, but the heap that is returned can only be used from a single thread because it is not thread safe.
By default the heaplets use a lock free singly linked list to keep track of rows that have been freed. This requires an atomic operation for each allocation and for each free. The scanning allocator uses an alternative approach. When a row is freed the row header is marked, and a row is allocated by scanning through the heaplet for rows that have been marked as free. Scanning uses atomic store and get, rather than more expensive synchronized atomic operations, so is generally faster than the linked list - provided a free row is found fairly quickly.
The scanning heaps have better best-case performance, but worse worse-case performance (if large numbers of rows need to be scanned before a free row is found). The best-case tends to be true if only one thread/activity is accessing a particular heap, and the worse-case if multiple activities are accessing a heap, particularly if the rows are being buffered. It is the default for thor which tends to have few active allocators, but not for roxie, which tends to have large numbers of allocators.
This is another varation on the scanning allocator, which further reduces the number of atomic operations. Usually when a row is allocated the link count on the heaplet is increased, and when it is freed the link count is decremented. This option delays decrementing the link count when the row is released, by marking the row with a different free flag. If it is subsequently reallocated there is no need to increment the link count. The downside is that it is more expensive to check whether a heaplet is completely empty (since you can no longer rely on the heaplet linkcount alone).
Thor has additional requirements to roxie. In roxie, if a query exceeds its memory requirements then it is terminated. Thor needs to be able to spill rows and other memory to disk and continue. This is achieved by allowing any process that stores buffered rows to register a callback with the memory manager. When more memory is required these callbacks are called to free up memory, and allow the job to continue.
Each callback can specify a priority - lower priority callbacks are called first since they are assumed to have a lower cost associated with spilling. When more memory is required the callbacks are called in priority order until one of them succeeds. The can also be passed a flag to indicate it is critical to force them to free up as much memory as possible.
A callback cannot allocate any memory from the memory manager. If it does it is likely to deadlock.
You cannot allocate memory while holding a lock if that lock is also required by a callback.
Again this will cause deadlock. If it proves impossible you can use a try-lock primitive in the callback, but it means you won't be able to spill those rows.
If the heaps are fragmented it may be more efficient to repack the heaps than spill to disk.
If you're resizing a potentially big block of memory use the resize function with the callback.
Some of the memory allocations cover more than one "page" - e.g., arrays used to store blocks of rows. (These are called huge pages internally, not to be confused with operating system support for huge pages...) When one of these memory blocks needs to be expanded you need to be careful:
Allocating a new page, copying, updating the pointer (within a cs) and then freeing is safe. Unfortunately it may involve copying a large chunk of memory. It may also fail if there isn't memory for the new and old block, even if the existing block could have been expanded into an adjacent block.
You can't lock, call a resize routine and update the pointer because the resize routine may need to allocate a new memory block- that may trigger a callback, which could in turn deadlock trying to gain the lock. (The callback may be from another thread...)
Therefore the memory manager contains a call which allows you to resize a block, but with a callback which is used to atomically update the pointer so it always remains thread safe.
Occasionally you have processes which read a large number of rows and then filter them so only a few are still held in memory. Rows tend to be allocated in sequence through the heap pages, which can mean those few remaining rows are scattered over many pages. If they could all be moved to a single page it would free up a significant amount of memory.
The memory manager contains a function to pack a set of rows into a smaller number of pages: IRowManager->compactRows().
This works by iterating through each of the rows in a list. If the row belongs to a heap that could be compacted, and isn't part of a full heaplet, then the row is moved. Since subsequent rows tend to be allocated from the same heaplet this has the effect of compacting the rows.
Much of the time Thor doesn't uses full memory available to it. If you are running multiple Thor processes on the same machine you may want to configure the system so that each Thor has a private block of memory, but there is also a shared block of memory which can be used by whichever process needs it.
The ILargeMemCallback provides a mechanism to dynamically allocate more memory to a process as it requires it. This could potentially be done in stages rather than all or nothing.
(Currently unused as far as I know... the main problem is that borrowing memory needs to be coordinated.)
When OS processes use a large amount of memory, mapping virtual addresses to physical addresses can begin to take a significant proportion of the execution time. This is especially true once the TLB is not large enough to store all the mappings. Huge pages can significantly help with this problem by reducing the number of TLB entries needed to cover the virtual address space. The memory manager supports huge pages in two different ways:
Huge pages can be preallocated (e.g., with hugeadm) for exclusive use as huge pages. If huge pages are enabled for a particular engine, and sufficient huge pages are available to supply the memory for the memory manager, then they will be used.
Linux kernels from 2.6.38 onward have support for transparent huge pages. These do not need to be preallocated, instead the operating system tries to use them behind the scenes. HPCC version 5.2 and following takes advantage of this feature to significantly speed memory access up when large amounts of memory are used by each process.
Preallocated huge pages tend to be more efficient, but they have the disadvantage that the operating system currently does not reuse unused huge pages for other purposes e.g., disk cache.
There is also a memory manager option to not return the memory to the operating system when it is no longer required. This has the advantage of not clearing the memory whenever it is required again, but the same disadvantage as preallocated huge pages that the unused memory cannot be used for disk cache. We recommend this option is selected when preallocated huge pages are in use - until the kernel allows them to be reused.
Changes in 6.x allow Thor to run multiple channels within the same process. This allows data that is constant for all channels to be shared between all slave channels - a prime example is the rhs of a lookup join. For the queries to run efficiently the memory manager needs to ensure that each slave channel has the same amount of memory - especially when memory is being used that is shared between them.
createGlobalRowManager() allows a single global row manager to be created which also provides slave row managers for the different channels via the querySlaveRowManager(unsigned slave) method.
',61)]))}const u=a(r,[["render",s]]);export{p as __pageData,u as default};
diff --git a/assets/devdoc_Metrics.md.BVV7YlV-.js b/assets/devdoc_Metrics.md.BVV7YlV-.js
new file mode 100644
index 00000000000..e83404d25fc
--- /dev/null
+++ b/assets/devdoc_Metrics.md.BVV7YlV-.js
@@ -0,0 +1,17 @@
+import{_ as t,c as i,a3 as s,o as a}from"./chunks/framework.DkhCEVKm.js";const m=JSON.parse('{"title":"Metrics Framework Design","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/Metrics.md","filePath":"devdoc/Metrics.md","lastUpdated":1731340314000}'),n={name:"devdoc/Metrics.md"};function r(o,e,l,h,c,p){return a(),i("div",null,e[0]||(e[0]=[s(`
This document describes the design of a metrics framework that allows HPCC Systems components to implement a metric collection strategy. Metrics provide the following functionality:
Alerts and monitoring
An important DevOps function is to monitor the cluster and providing alerts when problems are detected. Aggregated metric values from multiple sources provide the necessary data to build a complete picture of cluster health that drives monitoring and alerts.
Scaling
As described above, aggregated metric data is also used to dynamically respond to changing cluster demands and load. Metrics provide the monitoring capability to react and take action
Fault diagnosis and resource monitoring
Metrics provide historical data useful in diagnosing problems by profiling how demand and usage patterns may change prior to a fault. Predictive analysis can also be applied.
Analysis of jobs/workunits and profiling
With proper instrumentation, a robust dynamic metric strategy can track workunit processing. Internal problems with queries should be diagnosed from deep drill down logging.
The document consists of several sections in order to provide requirements as well as the design of framework components.
: A measurement defined by a component that represents an internal state that is useful in a system reliability engineering function. In the context of the framework, a metric is an object representing the above.
Metric Value
: The current value of a metric.
Metric Updating
: The component task of updating metric state.
Collection
: A framework process of selecting relevant metrics based on configuration and then retrieving their values.
Reporting
: A framework process of converting values obtained during a collection into a format suitable for ingestion by a collection system.
Trigger
: What causes the collection of metric values.
Collection System
: The store for metric values generated during the reporting framework process.
Roxie desires to keep a count of many different internal values. Some examples are
Disk type operations such as seeks and reads
Execution totals
Need to track items such as total numbers of items such as success and failures as well as breaking some counts into individual reasons. For example, failures may need be categorized such as as
Busy
Timeout
Bad input
Or even by priority (high, low, sla, etc.)
Current operational levels such as the length of internal queues
The latency of operations such as queue results, agent responses, and gateway responses
Roxie also has the need to track internal memory usage beyond the pod/system level capabilities. Tracking the state of its large fixed memory pool is necessary.
The Roxie buddy system also must track how often and who is completing requests. The "I Beat You To It" set of metrics must be collected and exposed in order to detect pending node failure. While specific action on these counts is not known up front, it appears clear that these values are useful and should be collected.
There does not appear to be a need for creating and destroying metrics dynamically. The set of metrics is most likely to be created at startup and remain active through the life of the Roxie. If, however, stats collection seeps into the metrics framework, dynamic creation and destruction of stats metrics is a likely requirement.
There are some interesting decisions with respect to ESP and collection of metrics. Different applications within ESP present different use cases for collection. Ownership of a given task drives some of these use cases. Take workunit queues. If ownership of the task, with respect to metrics, is WsWorkunits, then use cases are centric to that component. However, if agents listening on the queue are to report metrics, then a different set of use cases emerge. It is clear that additional work is needed to generate clear ownership of metrics gathered by ESP and/or the tasks it performs.
ESP needs to report the activeTransactions value from the TxSummary class(es). This gives an indication of how busy the ESP is in terms of client requests.
Direct measurement of response time in requests may not be useful since the type of request causes different execution paths within ESP that are expected to take widely varying amounts of time. Creation of metrics for each method is not recommended. However, two possible solutions are to a) create a metric for request types, or b) use a histogram to measure response time ranges. Another option mentioned redefines the meaning of a bucket in a histogram. Instead of a numeric distribution, each bucket represents a unique subtask within an overall "metric" representing a measured operation. This should be explored whether for operational or developmental purposes.
For tracking specific queries and their health, the feeling is that logging can accomplish this better than metrics since the list of queries to monitor will vary between clusters. Additionally, operational metrics solving the cases mentioned above will give a view into the overall health of ESP which will affect the execution of queries. Depending on actions taken by these metrics, scaling may solve overload conditions to keep cluster responsiveness acceptable.
For Roxie a workunit operates as a service. Measuring service performance using a histogram to capture response times as a distribution may be appropriate. Extracting the 95th percentile of response time may be useful as well.
There are currently no use cases requiring consistency between values of different metrics.
At this time the only concrete metric identified is the number of requests received. As the framework design progresses and ESP is instrumented, the list will grow.
This section covers the design and architecture of the framework. It discusses the main areas of the design, the interactions between each area, and an overall process model of how the framework operates.
The framework consists of three major areas: metrics, sinks, and the glue logic. These areas work together with the platform and the component to provide a reusable metrics collection function.
Metrics represent the quantifiable component state measurements used to track and assess the status of the component. Metrics are typically scalar values that are easily aggregated by a collection system. Aggregated values provide the necessary input to take component and cluster actions such as scaling up and down. The component is responsible for creating metrics and instrumenting the code. The framework provides the support for collecting and reporting the values. Metrics provide the following:
Simple methods for the component to update the metric
Simple methods for the framework to retrieve metric value(s)
Handling of all synchronization between updating and retrieving metric values
In addition, the framework provides the support for retrieving values so that the component does not participate in metric reporting. The component simply creates the metrics it needs, then instruments the component to update the metric whenever its state changes. For example, the component may create a metric that counts the total number of requests received. Then, wherever the component receives a request, a corresponding update to the count is added. Nowhere in the component is any code added to retrieve the count as that is handled by the framework.
Sinks provide a pluggable interface to hide the specifics of collection systems so that the metrics framework is independent of those dependencies. Sinks:
Operate independently of other sinks in the system
Convert metric native values into collection system specific measurements and reports
Drive the collection and reporting processes
The third area of the framework is the glue logic, referred to as the MetricsManager. It manages the metrics system for the component. It provides the following:
Handles framework initialization
Loads sinks as required
Manages the list of metrics for the component
Handles collection and reporting with a set of convenience methods used by sinks
The framework is designed to be instantiated into a component as part of its process and address space. All objects instantiated as part of the framework are owned by the component and are not shareable with any other component whether local or remote. Any coordination or consistency requirements that may arise in the implementation of a sink shall be the sole responsibility of the sink.
Components use metrics to measure their internal state. Metrics can represent everything from the number of requests received to the average length some value remains cached. Components are responsible for creating and updating metrics for each measured state. The framework shall provide a set of metrics designed to cover the majority of component measurement requirements. All metrics share a common interface to allow the framework to manage them in a common way.
To meet the requirement to manage metrics independent of the underlying metric state, all metrics implement a common interface. All metrics then add their specific methods to update and retrieve internal state. Generally the component uses the update method(s) to update state and the framework uses retrieval methods to get current state when reporting. The metric insures synchronized access.
For components that already have an implementation that tracks a metric, the framework provides a way to instantiate a custom metric. The custom metric allows the component to leverage the existing implementation and give the framework access to the metric value for collection and reporting. Note that custom metrics only support simple scalar metrics such as a counter or a gauge.
The framework defines a sink interface to support the different requirements of collection systems. Examples of collection systems are Prometheus, Datadog, and Elasticsearch. Each has different requirements for how and when measurements are ingested. The following are examples of different collection system requirements:
Polled vs Periodic
Single measurement vs multiple reports
Report format (JSON, text, etc.)
Push vs Pull
Sinks are responsible for two main functions: initiating a collection and reporting measurements to the collection system. The Metrics Reporter provides the support to complete these functions.
The sink encapsulates all of the collection system requirements providing a pluggable architecture that isolates components from these differences. The framework supports multiple sinks concurrently, each operating independently.
Instrumented components are not aware of the sink or sinks in use. Sinks can be changed without requiring changes to a component. Therefore, components are independent of the collection system(s) in use.
The metrics reporter class provides all of the common functions to bind together the component, the metrics it creates, and the sinks to which measurements are reported. It is responsible for the following:
Initialization of the framework
Managing the metrics created by the component
Handling collection and reporting as directed by configured sinks
A counter metric is a monotonically increasing value that "counts" the total occurrences of some event. Examples include the number of requests received, or the number of cache misses. Once created, the component instruments the code with updates to the count whenever appropriate.
A gauge metric is a continuously updated value representing the current state of an interesting value in the component. For example, the amount of memory used in an internal buffer, or the number of requests waiting on a queue. A gauge metric may increase or decrease in value as needed. Reading the value of a gauge is a stateless operation in that there are no dependencies on the previous reading. The value returned shall always be the current state.
Once created, the component shall update the gauge anytime the state of what is measured is updated. The metric shall provide methods to increase and decrease the value. The sink reads the value during collection and reporting.
A custom metric is a class that allows a component to leverage existing metrics. The component creates an instance of a custom metric (a templated class) and passes a reference to the underlying metric value. When collection is performed, the custom metric simply reads the value of the metric using the reference provided during construction. The component maintains full responsibility for updating the metric value as the custom metric class provides no update methods. The component is also responsible for ensuring atomic access to the value if necessary.
Records counts of measurements according to defined bucket limits. When created, the caller defines as set of bucket limits. During event recording, the component records measurements. The metric separates each recorded measurement into its bucket by testing the measurement value against each bucket limit using a less than or equal test. Each bucket contains a count of measurements meeting that criteria. Additionally, the metric maintains a default bucket for measurements outside of the maximum bucket limit. This is sometimes known as the "inf" bucket.
Some storage systems, such as Prometheus, require each bucket to accumulate its measurements with the previous bucket(s). It is the responsibility of the sink to accumulate values as needed.
A histogram metric that allows setting the bucket limit units in one domain, but take measurements in another domain. For example, the bucket limits may represent millisecond durations, yet it is more effecient to use execution cycles to take the measurements. A scaled histogram converts from the the measurement domain (cycles) to the limit units domain using a scale factor provided at initialization. All conversions are encapsulated in the scaled histogram class such that no external scaling is required by any consumer such as a sink.
This section discusses configuration. Since Helm charts are capable of combining configuration data at a global level into a component's specific configuration, The combined configuration takes the form as shown below. Note that as the design progresses it is expected that there will be additions.
Where (based on being a child of the current component):
metrics
: Metrics configuration for the component
metrics.sinks
: List of sinks defined for the component (may have been combined with global config)
metrics.sinks[].type
: The type for the sink. The type is substituted into the following pattern to determine the lib to load: libhpccmetrics<type><shared_object_extension>
metrics.sinks[].name
: A name for the sink.
metrics.sinks[].settings
: A set of key/value pairs passed to the sink when initialized. It should contain information necessary for the operation of the sink. Nested YML is supported. Example settings are the prometheus server name, or the collection period for a periodic sink.
Metric names shall follow a convention as outlined in this section. Because different collection systems have different requirements for how metric value reports are generated, naming is split into two parts.
First, each metric is given a base name that describes what the underlying value is. Second, meta data is assigned to each metric to further qualify the value. For example, a set of metrics may count the number of requests a component has received. Each metric would have the same base name, but meta data would separate types of request (GET vs POST), or disposition such as pass or fail.
Meta data further qualifies a metric value. This allows metrics to have the same name, but different scopes or categories. Generally, meta data is only used to furher qualify metrics that would have the same base name, but need further distinction. An example best describes a use case for meta data. Consider a component that accepts HTTP requests, but needs to track GET and POST requests separately. Instead of defining metrics with names post_requests.received and get_requests.received, the component creates two metrics with the base name requests.received and attaches meta data describing the request type of POST to one and GET to the other.
Use of meta data allows aggregating both types of requests into a single combined count of received requests while allowing a breakdown by type.
Meta data is represented as a key/value pair and is attached to the metric by the component during metric creation. The sink is responsible for converting meta data into useful information for the collection system during reporting.
The Component Instrumentation section covers how meta data is added to a metric.
In order to instrument a component for metrics using the framework, a component must include the metrics header from jlib (jmetrics.hpp) and add jlib as a dependent lib (if not already doing so).
The general steps for instrumentation are
Create a metrics reporter object
Create metric objects for each internal state to measure and add each to the reporter
Add updates to each metric throughout the component wherever metric state changes
The metrics reporter is a singleton created using the platform defined singleton pattern template. The component must obtain a reference to the reporter. Use the following example:
cpp
using namespace hpccMetrics;
+MetricsManager &metricsManager = queryMetricsManager();
Metrics are wrapped by a standard C++ shared pointer. The component is responsible for maintaining a reference to each shared pointer during the lifetime of the metric. The framework keeps a weak pointer to each metric and thus does not maintain a reference. The following is an example of creating a counter metric and adding it to the reporter. The using namespace eliminates the need to prefix all metrics types with hpccMetrics. Its use is assumed for all code examples that follow.
Note the metric type for both the shared pointer variable and in the make_shared template that creates the metric and returns a shared pointer. Simply substitute other metric types and handle any differences in the constructor arguments as needed.
Once created, add updates to the metric state throughout the component code where required. Using the above example, the following line of code increments the counter metric by 1.
cpp
pCounter->inc(1);
Note that only a single line of code is required to update the metric.
That's it! There are no component requirements related to collection or reporting of metric values. That is handled by the framework and loaded sinks.
For convenience, there are function templates that handle creating the reporter, creating a metric, and adding the metric to the reporter. For example, the above three lines of code that created the reporter, a metric, and added it, can be replaced by the following:
auto pCount = createMetricAndAddToManager<CounterMetric>("metricName", "description");
+
For convenience a similar function template exists for creating custom metrics. For a custom metric the framework must know the metric type and have a reference to the underlying state variable. The following template function handles creating a custom metric and adding it to the reporter (which is created if needed as well):
auto pCustomMetric = createCustomMetricAndAddToManager("customName", "description", metricType, value);
+
Where:
metricType
A defined metric type as defined by the MetricType enum.
value
A reference to the underlying event state which must be a scalar value convertable to a 64bit unsigned integer (__uint64)
A component, depending on requirements, may attach meta data to further qualify created metrics. Meta data takes the form of key value pairs. The base metric class MetricBase constructor defines a parameter for a vector of meta data. Metric subclasses also define meta data as a constructor parameter, however an empty vector is the default. The IMetric interface defines a method for retrieving the meta data.
Meta data is order dependent.
Below are two examples of constructing a metric with meta data. One creates the vector and passes it as a parameter, the other constructs the vector in place.
Metric units are treated separately from the base name and meta data. The reason is to allow the sink to translate based on collection system requirements. The base framework provides a convenience method for converting units into a string. However, the sink is free to do any conversions, both actual units and the string representation, as needed.
Metric units are defined using a subset of the StaticsMeasure enumeration values defined in jstatscodes.h. The current values are used:
SMeasureTimeNs - A time measurement in nanoseconds
SMeasureCount - A count of events
SMeasureSize - Size in bytes
`,132)]))}const u=t(n,[["render",r]]);export{m as __pageData,u as default};
diff --git a/assets/devdoc_Metrics.md.BVV7YlV-.lean.js b/assets/devdoc_Metrics.md.BVV7YlV-.lean.js
new file mode 100644
index 00000000000..e83404d25fc
--- /dev/null
+++ b/assets/devdoc_Metrics.md.BVV7YlV-.lean.js
@@ -0,0 +1,17 @@
+import{_ as t,c as i,a3 as s,o as a}from"./chunks/framework.DkhCEVKm.js";const m=JSON.parse('{"title":"Metrics Framework Design","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/Metrics.md","filePath":"devdoc/Metrics.md","lastUpdated":1731340314000}'),n={name:"devdoc/Metrics.md"};function r(o,e,l,h,c,p){return a(),i("div",null,e[0]||(e[0]=[s(`
This document describes the design of a metrics framework that allows HPCC Systems components to implement a metric collection strategy. Metrics provide the following functionality:
Alerts and monitoring
An important DevOps function is to monitor the cluster and providing alerts when problems are detected. Aggregated metric values from multiple sources provide the necessary data to build a complete picture of cluster health that drives monitoring and alerts.
Scaling
As described above, aggregated metric data is also used to dynamically respond to changing cluster demands and load. Metrics provide the monitoring capability to react and take action
Fault diagnosis and resource monitoring
Metrics provide historical data useful in diagnosing problems by profiling how demand and usage patterns may change prior to a fault. Predictive analysis can also be applied.
Analysis of jobs/workunits and profiling
With proper instrumentation, a robust dynamic metric strategy can track workunit processing. Internal problems with queries should be diagnosed from deep drill down logging.
The document consists of several sections in order to provide requirements as well as the design of framework components.
: A measurement defined by a component that represents an internal state that is useful in a system reliability engineering function. In the context of the framework, a metric is an object representing the above.
Metric Value
: The current value of a metric.
Metric Updating
: The component task of updating metric state.
Collection
: A framework process of selecting relevant metrics based on configuration and then retrieving their values.
Reporting
: A framework process of converting values obtained during a collection into a format suitable for ingestion by a collection system.
Trigger
: What causes the collection of metric values.
Collection System
: The store for metric values generated during the reporting framework process.
Roxie desires to keep a count of many different internal values. Some examples are
Disk type operations such as seeks and reads
Execution totals
Need to track items such as total numbers of items such as success and failures as well as breaking some counts into individual reasons. For example, failures may need be categorized such as as
Busy
Timeout
Bad input
Or even by priority (high, low, sla, etc.)
Current operational levels such as the length of internal queues
The latency of operations such as queue results, agent responses, and gateway responses
Roxie also has the need to track internal memory usage beyond the pod/system level capabilities. Tracking the state of its large fixed memory pool is necessary.
The Roxie buddy system also must track how often and who is completing requests. The "I Beat You To It" set of metrics must be collected and exposed in order to detect pending node failure. While specific action on these counts is not known up front, it appears clear that these values are useful and should be collected.
There does not appear to be a need for creating and destroying metrics dynamically. The set of metrics is most likely to be created at startup and remain active through the life of the Roxie. If, however, stats collection seeps into the metrics framework, dynamic creation and destruction of stats metrics is a likely requirement.
There are some interesting decisions with respect to ESP and collection of metrics. Different applications within ESP present different use cases for collection. Ownership of a given task drives some of these use cases. Take workunit queues. If ownership of the task, with respect to metrics, is WsWorkunits, then use cases are centric to that component. However, if agents listening on the queue are to report metrics, then a different set of use cases emerge. It is clear that additional work is needed to generate clear ownership of metrics gathered by ESP and/or the tasks it performs.
ESP needs to report the activeTransactions value from the TxSummary class(es). This gives an indication of how busy the ESP is in terms of client requests.
Direct measurement of response time in requests may not be useful since the type of request causes different execution paths within ESP that are expected to take widely varying amounts of time. Creation of metrics for each method is not recommended. However, two possible solutions are to a) create a metric for request types, or b) use a histogram to measure response time ranges. Another option mentioned redefines the meaning of a bucket in a histogram. Instead of a numeric distribution, each bucket represents a unique subtask within an overall "metric" representing a measured operation. This should be explored whether for operational or developmental purposes.
For tracking specific queries and their health, the feeling is that logging can accomplish this better than metrics since the list of queries to monitor will vary between clusters. Additionally, operational metrics solving the cases mentioned above will give a view into the overall health of ESP which will affect the execution of queries. Depending on actions taken by these metrics, scaling may solve overload conditions to keep cluster responsiveness acceptable.
For Roxie a workunit operates as a service. Measuring service performance using a histogram to capture response times as a distribution may be appropriate. Extracting the 95th percentile of response time may be useful as well.
There are currently no use cases requiring consistency between values of different metrics.
At this time the only concrete metric identified is the number of requests received. As the framework design progresses and ESP is instrumented, the list will grow.
This section covers the design and architecture of the framework. It discusses the main areas of the design, the interactions between each area, and an overall process model of how the framework operates.
The framework consists of three major areas: metrics, sinks, and the glue logic. These areas work together with the platform and the component to provide a reusable metrics collection function.
Metrics represent the quantifiable component state measurements used to track and assess the status of the component. Metrics are typically scalar values that are easily aggregated by a collection system. Aggregated values provide the necessary input to take component and cluster actions such as scaling up and down. The component is responsible for creating metrics and instrumenting the code. The framework provides the support for collecting and reporting the values. Metrics provide the following:
Simple methods for the component to update the metric
Simple methods for the framework to retrieve metric value(s)
Handling of all synchronization between updating and retrieving metric values
In addition, the framework provides the support for retrieving values so that the component does not participate in metric reporting. The component simply creates the metrics it needs, then instruments the component to update the metric whenever its state changes. For example, the component may create a metric that counts the total number of requests received. Then, wherever the component receives a request, a corresponding update to the count is added. Nowhere in the component is any code added to retrieve the count as that is handled by the framework.
Sinks provide a pluggable interface to hide the specifics of collection systems so that the metrics framework is independent of those dependencies. Sinks:
Operate independently of other sinks in the system
Convert metric native values into collection system specific measurements and reports
Drive the collection and reporting processes
The third area of the framework is the glue logic, referred to as the MetricsManager. It manages the metrics system for the component. It provides the following:
Handles framework initialization
Loads sinks as required
Manages the list of metrics for the component
Handles collection and reporting with a set of convenience methods used by sinks
The framework is designed to be instantiated into a component as part of its process and address space. All objects instantiated as part of the framework are owned by the component and are not shareable with any other component whether local or remote. Any coordination or consistency requirements that may arise in the implementation of a sink shall be the sole responsibility of the sink.
Components use metrics to measure their internal state. Metrics can represent everything from the number of requests received to the average length some value remains cached. Components are responsible for creating and updating metrics for each measured state. The framework shall provide a set of metrics designed to cover the majority of component measurement requirements. All metrics share a common interface to allow the framework to manage them in a common way.
To meet the requirement to manage metrics independent of the underlying metric state, all metrics implement a common interface. All metrics then add their specific methods to update and retrieve internal state. Generally the component uses the update method(s) to update state and the framework uses retrieval methods to get current state when reporting. The metric insures synchronized access.
For components that already have an implementation that tracks a metric, the framework provides a way to instantiate a custom metric. The custom metric allows the component to leverage the existing implementation and give the framework access to the metric value for collection and reporting. Note that custom metrics only support simple scalar metrics such as a counter or a gauge.
The framework defines a sink interface to support the different requirements of collection systems. Examples of collection systems are Prometheus, Datadog, and Elasticsearch. Each has different requirements for how and when measurements are ingested. The following are examples of different collection system requirements:
Polled vs Periodic
Single measurement vs multiple reports
Report format (JSON, text, etc.)
Push vs Pull
Sinks are responsible for two main functions: initiating a collection and reporting measurements to the collection system. The Metrics Reporter provides the support to complete these functions.
The sink encapsulates all of the collection system requirements providing a pluggable architecture that isolates components from these differences. The framework supports multiple sinks concurrently, each operating independently.
Instrumented components are not aware of the sink or sinks in use. Sinks can be changed without requiring changes to a component. Therefore, components are independent of the collection system(s) in use.
The metrics reporter class provides all of the common functions to bind together the component, the metrics it creates, and the sinks to which measurements are reported. It is responsible for the following:
Initialization of the framework
Managing the metrics created by the component
Handling collection and reporting as directed by configured sinks
A counter metric is a monotonically increasing value that "counts" the total occurrences of some event. Examples include the number of requests received, or the number of cache misses. Once created, the component instruments the code with updates to the count whenever appropriate.
A gauge metric is a continuously updated value representing the current state of an interesting value in the component. For example, the amount of memory used in an internal buffer, or the number of requests waiting on a queue. A gauge metric may increase or decrease in value as needed. Reading the value of a gauge is a stateless operation in that there are no dependencies on the previous reading. The value returned shall always be the current state.
Once created, the component shall update the gauge anytime the state of what is measured is updated. The metric shall provide methods to increase and decrease the value. The sink reads the value during collection and reporting.
A custom metric is a class that allows a component to leverage existing metrics. The component creates an instance of a custom metric (a templated class) and passes a reference to the underlying metric value. When collection is performed, the custom metric simply reads the value of the metric using the reference provided during construction. The component maintains full responsibility for updating the metric value as the custom metric class provides no update methods. The component is also responsible for ensuring atomic access to the value if necessary.
Records counts of measurements according to defined bucket limits. When created, the caller defines as set of bucket limits. During event recording, the component records measurements. The metric separates each recorded measurement into its bucket by testing the measurement value against each bucket limit using a less than or equal test. Each bucket contains a count of measurements meeting that criteria. Additionally, the metric maintains a default bucket for measurements outside of the maximum bucket limit. This is sometimes known as the "inf" bucket.
Some storage systems, such as Prometheus, require each bucket to accumulate its measurements with the previous bucket(s). It is the responsibility of the sink to accumulate values as needed.
A histogram metric that allows setting the bucket limit units in one domain, but take measurements in another domain. For example, the bucket limits may represent millisecond durations, yet it is more effecient to use execution cycles to take the measurements. A scaled histogram converts from the the measurement domain (cycles) to the limit units domain using a scale factor provided at initialization. All conversions are encapsulated in the scaled histogram class such that no external scaling is required by any consumer such as a sink.
This section discusses configuration. Since Helm charts are capable of combining configuration data at a global level into a component's specific configuration, The combined configuration takes the form as shown below. Note that as the design progresses it is expected that there will be additions.
Where (based on being a child of the current component):
metrics
: Metrics configuration for the component
metrics.sinks
: List of sinks defined for the component (may have been combined with global config)
metrics.sinks[].type
: The type for the sink. The type is substituted into the following pattern to determine the lib to load: libhpccmetrics<type><shared_object_extension>
metrics.sinks[].name
: A name for the sink.
metrics.sinks[].settings
: A set of key/value pairs passed to the sink when initialized. It should contain information necessary for the operation of the sink. Nested YML is supported. Example settings are the prometheus server name, or the collection period for a periodic sink.
Metric names shall follow a convention as outlined in this section. Because different collection systems have different requirements for how metric value reports are generated, naming is split into two parts.
First, each metric is given a base name that describes what the underlying value is. Second, meta data is assigned to each metric to further qualify the value. For example, a set of metrics may count the number of requests a component has received. Each metric would have the same base name, but meta data would separate types of request (GET vs POST), or disposition such as pass or fail.
Meta data further qualifies a metric value. This allows metrics to have the same name, but different scopes or categories. Generally, meta data is only used to furher qualify metrics that would have the same base name, but need further distinction. An example best describes a use case for meta data. Consider a component that accepts HTTP requests, but needs to track GET and POST requests separately. Instead of defining metrics with names post_requests.received and get_requests.received, the component creates two metrics with the base name requests.received and attaches meta data describing the request type of POST to one and GET to the other.
Use of meta data allows aggregating both types of requests into a single combined count of received requests while allowing a breakdown by type.
Meta data is represented as a key/value pair and is attached to the metric by the component during metric creation. The sink is responsible for converting meta data into useful information for the collection system during reporting.
The Component Instrumentation section covers how meta data is added to a metric.
In order to instrument a component for metrics using the framework, a component must include the metrics header from jlib (jmetrics.hpp) and add jlib as a dependent lib (if not already doing so).
The general steps for instrumentation are
Create a metrics reporter object
Create metric objects for each internal state to measure and add each to the reporter
Add updates to each metric throughout the component wherever metric state changes
The metrics reporter is a singleton created using the platform defined singleton pattern template. The component must obtain a reference to the reporter. Use the following example:
cpp
using namespace hpccMetrics;
+MetricsManager &metricsManager = queryMetricsManager();
Metrics are wrapped by a standard C++ shared pointer. The component is responsible for maintaining a reference to each shared pointer during the lifetime of the metric. The framework keeps a weak pointer to each metric and thus does not maintain a reference. The following is an example of creating a counter metric and adding it to the reporter. The using namespace eliminates the need to prefix all metrics types with hpccMetrics. Its use is assumed for all code examples that follow.
Note the metric type for both the shared pointer variable and in the make_shared template that creates the metric and returns a shared pointer. Simply substitute other metric types and handle any differences in the constructor arguments as needed.
Once created, add updates to the metric state throughout the component code where required. Using the above example, the following line of code increments the counter metric by 1.
cpp
pCounter->inc(1);
Note that only a single line of code is required to update the metric.
That's it! There are no component requirements related to collection or reporting of metric values. That is handled by the framework and loaded sinks.
For convenience, there are function templates that handle creating the reporter, creating a metric, and adding the metric to the reporter. For example, the above three lines of code that created the reporter, a metric, and added it, can be replaced by the following:
auto pCount = createMetricAndAddToManager<CounterMetric>("metricName", "description");
+
For convenience a similar function template exists for creating custom metrics. For a custom metric the framework must know the metric type and have a reference to the underlying state variable. The following template function handles creating a custom metric and adding it to the reporter (which is created if needed as well):
auto pCustomMetric = createCustomMetricAndAddToManager("customName", "description", metricType, value);
+
Where:
metricType
A defined metric type as defined by the MetricType enum.
value
A reference to the underlying event state which must be a scalar value convertable to a 64bit unsigned integer (__uint64)
A component, depending on requirements, may attach meta data to further qualify created metrics. Meta data takes the form of key value pairs. The base metric class MetricBase constructor defines a parameter for a vector of meta data. Metric subclasses also define meta data as a constructor parameter, however an empty vector is the default. The IMetric interface defines a method for retrieving the meta data.
Meta data is order dependent.
Below are two examples of constructing a metric with meta data. One creates the vector and passes it as a parameter, the other constructs the vector in place.
Metric units are treated separately from the base name and meta data. The reason is to allow the sink to translate based on collection system requirements. The base framework provides a convenience method for converting units into a string. However, the sink is free to do any conversions, both actual units and the string representation, as needed.
Metric units are defined using a subset of the StaticsMeasure enumeration values defined in jstatscodes.h. The current values are used:
SMeasureTimeNs - A time measurement in nanoseconds
SMeasureCount - A count of events
SMeasureSize - Size in bytes
`,132)]))}const u=t(n,[["render",r]]);export{m as __pageData,u as default};
diff --git a/assets/devdoc_NewFileProcessing.md.DGLPSf2v.js b/assets/devdoc_NewFileProcessing.md.DGLPSf2v.js
new file mode 100644
index 00000000000..064d17d7d1b
--- /dev/null
+++ b/assets/devdoc_NewFileProcessing.md.DGLPSf2v.js
@@ -0,0 +1,14 @@
+import{_ as t,c as a,a3 as i,o}from"./chunks/framework.DkhCEVKm.js";const f=JSON.parse('{"title":"Storage planes","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/NewFileProcessing.md","filePath":"devdoc/NewFileProcessing.md","lastUpdated":1731340314000}'),r={name:"devdoc/NewFileProcessing.md"};function s(n,e,l,p,h,c){return o(),a("div",null,e[0]||(e[0]=[i(`
Documentation about the new file work.
YAML files. The following are the YAML definitions which are used to serialize file information from dali/external store to the engines and if necessary to the worker nodes.
This is already covered in the deployed helm charts. It has been extended and rationalized slightly.
storage:
: hostGroups: - name: <required> hosts: [ .... ] - name: <required> hostGroup: <name> count: <unsigned:#hosts> # how many hosts within the host group are used ?(default is number of hosts) offset: <unsigned:0> # index of first host included in the derived group delta: <unsigned:0> # first host within the range[offset..offset+count-1] in the derived group
planes:
+
+: name: \\<required\\> prefix: \\<path\\> \\# Root directory for
+ accessing the plane (if pvc defined), or url to access plane.
+ numDevices: 1 \\# number of devices that are part of the plane
+ hostGroup: \\<name\\> \\# Name of the host group for bare metal
+ hosts: \\[ host-names \\] \\# A list of host names for bare metal
+ secret: \\<secret-id\\> \\# what secret is required to access the
+ files. options: \\# not sure if it is needed
+
Changes: * The replication information has been removed from the storage plane. It will now be specified on the thor instance indicating where (if anywhere) files are replicated. * The hash character (#) in a prefix or a secret name will be substituted with the device number. This replaces the old includeDeviceInPath property. This allows more flexible device substition for both local mounts and storage accounts. The number of hashes provides the default padding for the device number. (Existing Helm charts will need to be updated to follow these new rules.) * Neither thor or roxie replication is special cased. They are represented as multiple locations that the file lives (see examples below). Existing baremetal environments would be mapped to this new representation with implicit replication planes. (It is worth checking the mapping to roxie is fine.)
file: - name: <logical-file-name> format: <type> # e.g. flat, csv, xml, key, parquet meta: <binary> # (opt) format of the file, (serialized hpcc field information). metaCrc: <unsigned> # hash of the meta numParts # How many file parts. singlePartNoSuffix: <boolean> # Does a single part file include .part_1_of_1? numRows: # total number of rows in the file (if known) rawSize: # total uncompressed size diskSize # is this useful? when binary copying? planes: [] # list of storage planes that the file is stored on. tlk: # ???Should the tlk be stored in the meta and returned? splitType: <split-format> # Are there associated split points, and if so what format? (And if so what variant?)
#options relating to the format of the input file:
: grouped: <boolean> # is the file grouped? compressed: <boolean> blockCompressed: <boolean> formatOptions: # Any options that relate to the file format e.g. csvTerminator. These are nested because they can be completely free format recordSize: # if a fixed size record. Not really sure it is useful
part: \\# optional information about each of the file parts (Cannot
+implement virtual file position without this) - numRows: \\<count\\>
+\\# number of rows in the file part rawSize: \\<size\\> \\# uncompressed
+size of the file part diskSize: \\<size\\> \\# size of the part on disk
+
# extra fields that are used to return information from the file lookup service
missing: <boolean> # true if the file could not be found external: <boolean> # filename of the form external:: or plane:
If the information needs to be signed to be passed to dafilesrv for example, the entire structure of (storage, files) is serialized, and compressed, and that then signed.
Logically executed on the engine, and retrived from dali or in future versions from an esp service (even if for remote reads).
GetFileInfomation(<logical-filename>, <options>)
The logical-filename can be any logical name - including a super file, or an implicit superfile.
options include: * Are compressed sizes needed? * Are signatures required? * Is virtual fileposition (non-local) required? * name of the user
This returns a structure that provides information about a list of files
meta:
: hostGroups: storage: files: secrets: #The secret names are known, how do we know which keys are required for those secrets?
Some key questions: * Should the TLK be in the dali meta information? [Possibly, but not in first phase. ] * Should the split points be in the dali meta information? [Probably not, but the meta should indicate whether they exist, and if so what format they are. ] * Super files (implicit or explicit) can contain the same file information more than once. Should it be duplicated, or have a flag to indicate a repeat. [I suspect this is fairly uncommon, so duplication would be fine for the first version.] * What storage plane information is serialized back? [ all is simplest. Can optimize later. ]
NOTE: This doesn't address the question of writing to a disk file...
Local class for interpreting the results. Logically executed on the manager, and may gather extra information that will be serialized to all workers. The aim is that the same class implementations are used by all the engines (and fileview in esp).
MasterFileCollection : RemoteFileCollection : FileCollection(eclReadOptions, eclFormatOptions, wuid, user, expectedMeta, projectedMeta); MasterFileCollection //Master has access to dali RemoteFileCollection : has access to remote esp // think some more
FileCollection::GatherFileInformation(<logical-filename>, gatherOptions); - potentially called once per query. - class is responsible for optimizing case where it matches the previous call (e.g. in a child query). - possibly responsible for retrieving the split points ()
Following options are used to control whether split points are retrieved when file information is gathered * number of channels reading the data? * number of strands reading each channel? * preserve order?
gatherOptions: * is it a temporary file?
This class serializes all information to every worker, where it is used to recereate a copy of the master filecollection. This will contain information derived from dali, and locally e.g. options specified in the activity helper. Each worker has a complete copy of the file information. (This is similar to dafilesrv with security tokens.)
The files that are actually required by a worker are calculated by calling the following function. (Note the derived information is not serialized.)
Things to bear in mind: - Optimize same file reused in a child query (filter likely to change) - Optimize same format reused in a child query (filename may be dynamic) - Intergrating third party file formats and distributed file systems may require extra information. - optimize reusing the format options. - ideally fail over to a backup copy midstream.. and retry in failed read e.g. if network fault
: planes: #Simple 400 way thor - name: thor400 prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group #The storage plane used for replicating files on thor. - name: thor400_R1 prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group offset: 1 # A 200 way thor using the first 200 nodes as the thor 400 - name: thor200A prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group size: 200 # A 200 way thor using the second 200 nodes as the thor 400 - name: thor200B prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group size: 200 start: 200 # The replication plane for a 200 way thor using the second 200 nodes as the thor 400 - name: thor200B_R1 prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group size: 200 start: 200 offset: 1 # A roxie storage where 50way files are stored on a 100 way roxie - name: roxie100 prefix: /var/lib/HPCCSystems/roxie100 hosts: thor400Group size: 50 # The replica of the roxie storage where 50way files are stored on a 100 way roxie - name: roxie100_R1 prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group start: 50 size: 50
There is no special casing of roxie replication, and each file exists on multiple storage planes. All of these should be considered when determining which is the best copy to read from a particular engine node.
Creating storage planes from an existing systems [implemented]
b) [a] Start simplifying information in dali meta (e.g. partmask, remove full path name) c) [a] Switch reading code to use storageplane away from using dali path and environment paths - in ALL disk reading and writing code - change numDevices so it matches the container d) [c] Convert dali information from using copies to multiple groups/planese) [a] Reimplement the current code to create an IPropertyTree from dali file information (in a form that can be reused in dali) *f) [e] Refactor existing PR to use data in an IPropertyTree and cleanly separate the interfaces. g) Switch hthor over to using the new classes by default and work through all issues h) Refactor stream reading code. Look at the spark interfaces for inspiration/compatibility i) Refactor disk writing code into common class? j) [e] create esp service for accessing meta information k) [h] Refactor and review azure blob code l) [k] Re-implement S3 reading and writing code.
m) Switch fileview over to using the new classes. (Great test they can be used in another context + fixes a longstanding bug.)
) Implications for index reading? Will they end up being treated as a normal file? Don't implement for 8.0, although interface may support it.
Buffer sizes: - storage plane specifies an optimal reading minimum - compression may have a requirement - the use for the data may impose a requirement e.g. a subset of the data, or only fetching a single record
parallel disk reading may want to read a big chunk, but then process in sections. groan.
Look at lambda functions to create split points for a file. Can we use the java classes to implement it on binary files (and csv/xml)?
****************** Reading classes and meta information****************** meta comes from a combination of the information in dfs and the helper
The main meta information uses the same structure that is return by the function that returns file infromation from dali. The format specific options are contained in a nested attribute so they can be completely arbitrary
The helper class also generates a meta structure. Some options fill in root elements - e.g. compressed. Some fill in a new section (hints: @x=y). The format options are generated from the paramaters to the dataset format.
note normally there is only a single (or very few) files, so merging isn't too painful. queryMeta() queryOptions() rename meta to format? ???
Where does DFUserver fit in in a container system?
DFU has the following main functionality in a bare metal system: a) Spray a file from a 1 way landing zone to an N-way thor b) Convert file format when spraying. I suspect utf-16->utf8 is the only option actually used. c) Spray multiple files from a landing zone to a single logical file on an N-way thor d) Copy a logical file from a remote environment e) Despray a logical file to an external landing zone. f) Replicate an existing logical file on a given group. g) Copy logical files between groups h) File monitoring i) logical file operations j) superfile operations
ECL has the ability to read a logical file directly from a landingzone using 'FILE::<ip>' file syntax, but I don't think it is used very frequently.
How does this map to a containerized system? I think the same basic operations are likely to be useful. a) In most scenarios Landing zones are likely to be replaced with (blob) storage accounts. But for security reasons these are likely to remain distinct from the main location used by HPCC to store datasets. (The customer will have only access keys to copy files to and from those storage accounts.) The containerized system has a way for ECL to directly read from a blob storage account ('PLANE::<plane'), but I imagine users will still want to copy the files in many situations to control the lifetime of the copies etc. b) We still need a way to convert from utf16 to utf8, or extend the platform to allow utf16 to be read directly. c) This is still equally useful, allowing a set of files to be stored as a single file in a form that is easy for ECL to process. d) Important for copying data from an existing bare metal system to the cloud, and from a cloud system back to a bare metal system. e) Useful for exporting results to customers f+g) Essentially the same thing in the cloud world. It might still be useful to have h) I suspect we will need to map this to cloud-specific apis. i+j) Just as applicable in the container world.
Broadly, landing zones in bare metal map to special storage planes in containerized, and groups also map to more general storage planes.
There are a couple of complications connected with the implementation:
Copying is currently done by starting an ftslave process on either the source or the target nodes. In the container world there is no local node, and I think we would prefer not to start a process in order to copy each file. 2) Copying between storage groups should be done using the cloud provider api, rather than transferring data via a k8s job.
Suggestions:
Have a load balanced dafilesrv which supports multiple replicas. It would have a secure external service, and an internal service for trusted components.
Move the ftslave logic into dafilesrv. Move the current code for ftslave actions into dafilesrv with new operations.
When copying from/to a bare metal system the requests are sent to the dafilesrv for the node that currently runs ftslave. For a container system the requests are sent to the loadbalanced service.
It might be possible to migrate to lamda style functions for some of the work...
A later optimization would use a cloud service where it was possible.
When local split points are supported it may be better to spray a file 1:1 along with partition information. Even without local split points it may still be better to spray a file 1:1 (cheaper).
What are the spray targets? It may need to be storage plane + number of parts, rather than a target cluster. The default number of parts is the #devices on the storage plane.
=> Milestones a) Move ftslave code to dafilesrv (partition, pull, push) [Should be included in 7.12.x stream to allow remote read compatibility?] b) Create a dafilesrv component to the helm charts, with internal and external services. c) use storage planes to determine how files are sprayed etc. (bare-metal, #devices) Adapt dfu/fileservices calls to take (storageplane,number) instead of cluster. There should already be a 1:1 mapping from existing cluster to storage planes in a bare-metal system, so this may not involve much work. [May also need a flag to indicate if ._1_of_1 is appended?] d) Select correct dafilesrv for bare-metal storage planes, or load balanced service for other. (May need to think through how remote files are represented.)
=> Can import from a bare metal system or a containerized system using command line??
: NOTE: Bare-metal to containerized will likely need push operations on the bare-metal system. (And therefore serialized security information) This may still cause issues since it is unlikely containerized will be able to pull from bare-metal. Pushing, but not creating a logical file entry on the containerized system should be easier since it can use a local storage plane definition.
e) Switch over to using the esp based meta information, so that it can include details of storage planes and secrets.
: [Note this would also need to be in 7.12.x to allow remote export to containerized, that may well be a step too far]
f) Add option to configure the number of file parts for spray/copy/despray g) Ensure that eclwatch picks up the list of storage planes (and the default number of file parts), and has ability to specify #parts.
Later: h) plan how cloud-services can be used for some of the copies i) investigate using serverless functions to calculate split points. j) Use refactored disk read/write interfaces to clean up read and copy code. k) we may not want to expose access keys to allow remote reads/writes - in which they would need to be pushed from a bare-metal dafilesrv to a containerized dafilesrv.
Other dependencies: * Refactored file meta information. If this is switching to being plane based, then the meta information should also be plane based. Main difference is not including the path in the meta information (can just be ignored) * esp service for getting file information. When reading remotely it needs to go via this now...
`,80)]))}const m=t(r,[["render",s]]);export{f as __pageData,m as default};
diff --git a/assets/devdoc_NewFileProcessing.md.DGLPSf2v.lean.js b/assets/devdoc_NewFileProcessing.md.DGLPSf2v.lean.js
new file mode 100644
index 00000000000..064d17d7d1b
--- /dev/null
+++ b/assets/devdoc_NewFileProcessing.md.DGLPSf2v.lean.js
@@ -0,0 +1,14 @@
+import{_ as t,c as a,a3 as i,o}from"./chunks/framework.DkhCEVKm.js";const f=JSON.parse('{"title":"Storage planes","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/NewFileProcessing.md","filePath":"devdoc/NewFileProcessing.md","lastUpdated":1731340314000}'),r={name:"devdoc/NewFileProcessing.md"};function s(n,e,l,p,h,c){return o(),a("div",null,e[0]||(e[0]=[i(`
Documentation about the new file work.
YAML files. The following are the YAML definitions which are used to serialize file information from dali/external store to the engines and if necessary to the worker nodes.
This is already covered in the deployed helm charts. It has been extended and rationalized slightly.
storage:
: hostGroups: - name: <required> hosts: [ .... ] - name: <required> hostGroup: <name> count: <unsigned:#hosts> # how many hosts within the host group are used ?(default is number of hosts) offset: <unsigned:0> # index of first host included in the derived group delta: <unsigned:0> # first host within the range[offset..offset+count-1] in the derived group
planes:
+
+: name: \\<required\\> prefix: \\<path\\> \\# Root directory for
+ accessing the plane (if pvc defined), or url to access plane.
+ numDevices: 1 \\# number of devices that are part of the plane
+ hostGroup: \\<name\\> \\# Name of the host group for bare metal
+ hosts: \\[ host-names \\] \\# A list of host names for bare metal
+ secret: \\<secret-id\\> \\# what secret is required to access the
+ files. options: \\# not sure if it is needed
+
Changes: * The replication information has been removed from the storage plane. It will now be specified on the thor instance indicating where (if anywhere) files are replicated. * The hash character (#) in a prefix or a secret name will be substituted with the device number. This replaces the old includeDeviceInPath property. This allows more flexible device substition for both local mounts and storage accounts. The number of hashes provides the default padding for the device number. (Existing Helm charts will need to be updated to follow these new rules.) * Neither thor or roxie replication is special cased. They are represented as multiple locations that the file lives (see examples below). Existing baremetal environments would be mapped to this new representation with implicit replication planes. (It is worth checking the mapping to roxie is fine.)
file: - name: <logical-file-name> format: <type> # e.g. flat, csv, xml, key, parquet meta: <binary> # (opt) format of the file, (serialized hpcc field information). metaCrc: <unsigned> # hash of the meta numParts # How many file parts. singlePartNoSuffix: <boolean> # Does a single part file include .part_1_of_1? numRows: # total number of rows in the file (if known) rawSize: # total uncompressed size diskSize # is this useful? when binary copying? planes: [] # list of storage planes that the file is stored on. tlk: # ???Should the tlk be stored in the meta and returned? splitType: <split-format> # Are there associated split points, and if so what format? (And if so what variant?)
#options relating to the format of the input file:
: grouped: <boolean> # is the file grouped? compressed: <boolean> blockCompressed: <boolean> formatOptions: # Any options that relate to the file format e.g. csvTerminator. These are nested because they can be completely free format recordSize: # if a fixed size record. Not really sure it is useful
part: \\# optional information about each of the file parts (Cannot
+implement virtual file position without this) - numRows: \\<count\\>
+\\# number of rows in the file part rawSize: \\<size\\> \\# uncompressed
+size of the file part diskSize: \\<size\\> \\# size of the part on disk
+
# extra fields that are used to return information from the file lookup service
missing: <boolean> # true if the file could not be found external: <boolean> # filename of the form external:: or plane:
If the information needs to be signed to be passed to dafilesrv for example, the entire structure of (storage, files) is serialized, and compressed, and that then signed.
Logically executed on the engine, and retrived from dali or in future versions from an esp service (even if for remote reads).
GetFileInfomation(<logical-filename>, <options>)
The logical-filename can be any logical name - including a super file, or an implicit superfile.
options include: * Are compressed sizes needed? * Are signatures required? * Is virtual fileposition (non-local) required? * name of the user
This returns a structure that provides information about a list of files
meta:
: hostGroups: storage: files: secrets: #The secret names are known, how do we know which keys are required for those secrets?
Some key questions: * Should the TLK be in the dali meta information? [Possibly, but not in first phase. ] * Should the split points be in the dali meta information? [Probably not, but the meta should indicate whether they exist, and if so what format they are. ] * Super files (implicit or explicit) can contain the same file information more than once. Should it be duplicated, or have a flag to indicate a repeat. [I suspect this is fairly uncommon, so duplication would be fine for the first version.] * What storage plane information is serialized back? [ all is simplest. Can optimize later. ]
NOTE: This doesn't address the question of writing to a disk file...
Local class for interpreting the results. Logically executed on the manager, and may gather extra information that will be serialized to all workers. The aim is that the same class implementations are used by all the engines (and fileview in esp).
MasterFileCollection : RemoteFileCollection : FileCollection(eclReadOptions, eclFormatOptions, wuid, user, expectedMeta, projectedMeta); MasterFileCollection //Master has access to dali RemoteFileCollection : has access to remote esp // think some more
FileCollection::GatherFileInformation(<logical-filename>, gatherOptions); - potentially called once per query. - class is responsible for optimizing case where it matches the previous call (e.g. in a child query). - possibly responsible for retrieving the split points ()
Following options are used to control whether split points are retrieved when file information is gathered * number of channels reading the data? * number of strands reading each channel? * preserve order?
gatherOptions: * is it a temporary file?
This class serializes all information to every worker, where it is used to recereate a copy of the master filecollection. This will contain information derived from dali, and locally e.g. options specified in the activity helper. Each worker has a complete copy of the file information. (This is similar to dafilesrv with security tokens.)
The files that are actually required by a worker are calculated by calling the following function. (Note the derived information is not serialized.)
Things to bear in mind: - Optimize same file reused in a child query (filter likely to change) - Optimize same format reused in a child query (filename may be dynamic) - Intergrating third party file formats and distributed file systems may require extra information. - optimize reusing the format options. - ideally fail over to a backup copy midstream.. and retry in failed read e.g. if network fault
: planes: #Simple 400 way thor - name: thor400 prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group #The storage plane used for replicating files on thor. - name: thor400_R1 prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group offset: 1 # A 200 way thor using the first 200 nodes as the thor 400 - name: thor200A prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group size: 200 # A 200 way thor using the second 200 nodes as the thor 400 - name: thor200B prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group size: 200 start: 200 # The replication plane for a 200 way thor using the second 200 nodes as the thor 400 - name: thor200B_R1 prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group size: 200 start: 200 offset: 1 # A roxie storage where 50way files are stored on a 100 way roxie - name: roxie100 prefix: /var/lib/HPCCSystems/roxie100 hosts: thor400Group size: 50 # The replica of the roxie storage where 50way files are stored on a 100 way roxie - name: roxie100_R1 prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group start: 50 size: 50
There is no special casing of roxie replication, and each file exists on multiple storage planes. All of these should be considered when determining which is the best copy to read from a particular engine node.
Creating storage planes from an existing systems [implemented]
b) [a] Start simplifying information in dali meta (e.g. partmask, remove full path name) c) [a] Switch reading code to use storageplane away from using dali path and environment paths - in ALL disk reading and writing code - change numDevices so it matches the container d) [c] Convert dali information from using copies to multiple groups/planese) [a] Reimplement the current code to create an IPropertyTree from dali file information (in a form that can be reused in dali) *f) [e] Refactor existing PR to use data in an IPropertyTree and cleanly separate the interfaces. g) Switch hthor over to using the new classes by default and work through all issues h) Refactor stream reading code. Look at the spark interfaces for inspiration/compatibility i) Refactor disk writing code into common class? j) [e] create esp service for accessing meta information k) [h] Refactor and review azure blob code l) [k] Re-implement S3 reading and writing code.
m) Switch fileview over to using the new classes. (Great test they can be used in another context + fixes a longstanding bug.)
) Implications for index reading? Will they end up being treated as a normal file? Don't implement for 8.0, although interface may support it.
Buffer sizes: - storage plane specifies an optimal reading minimum - compression may have a requirement - the use for the data may impose a requirement e.g. a subset of the data, or only fetching a single record
parallel disk reading may want to read a big chunk, but then process in sections. groan.
Look at lambda functions to create split points for a file. Can we use the java classes to implement it on binary files (and csv/xml)?
****************** Reading classes and meta information****************** meta comes from a combination of the information in dfs and the helper
The main meta information uses the same structure that is return by the function that returns file infromation from dali. The format specific options are contained in a nested attribute so they can be completely arbitrary
The helper class also generates a meta structure. Some options fill in root elements - e.g. compressed. Some fill in a new section (hints: @x=y). The format options are generated from the paramaters to the dataset format.
note normally there is only a single (or very few) files, so merging isn't too painful. queryMeta() queryOptions() rename meta to format? ???
Where does DFUserver fit in in a container system?
DFU has the following main functionality in a bare metal system: a) Spray a file from a 1 way landing zone to an N-way thor b) Convert file format when spraying. I suspect utf-16->utf8 is the only option actually used. c) Spray multiple files from a landing zone to a single logical file on an N-way thor d) Copy a logical file from a remote environment e) Despray a logical file to an external landing zone. f) Replicate an existing logical file on a given group. g) Copy logical files between groups h) File monitoring i) logical file operations j) superfile operations
ECL has the ability to read a logical file directly from a landingzone using 'FILE::<ip>' file syntax, but I don't think it is used very frequently.
How does this map to a containerized system? I think the same basic operations are likely to be useful. a) In most scenarios Landing zones are likely to be replaced with (blob) storage accounts. But for security reasons these are likely to remain distinct from the main location used by HPCC to store datasets. (The customer will have only access keys to copy files to and from those storage accounts.) The containerized system has a way for ECL to directly read from a blob storage account ('PLANE::<plane'), but I imagine users will still want to copy the files in many situations to control the lifetime of the copies etc. b) We still need a way to convert from utf16 to utf8, or extend the platform to allow utf16 to be read directly. c) This is still equally useful, allowing a set of files to be stored as a single file in a form that is easy for ECL to process. d) Important for copying data from an existing bare metal system to the cloud, and from a cloud system back to a bare metal system. e) Useful for exporting results to customers f+g) Essentially the same thing in the cloud world. It might still be useful to have h) I suspect we will need to map this to cloud-specific apis. i+j) Just as applicable in the container world.
Broadly, landing zones in bare metal map to special storage planes in containerized, and groups also map to more general storage planes.
There are a couple of complications connected with the implementation:
Copying is currently done by starting an ftslave process on either the source or the target nodes. In the container world there is no local node, and I think we would prefer not to start a process in order to copy each file. 2) Copying between storage groups should be done using the cloud provider api, rather than transferring data via a k8s job.
Suggestions:
Have a load balanced dafilesrv which supports multiple replicas. It would have a secure external service, and an internal service for trusted components.
Move the ftslave logic into dafilesrv. Move the current code for ftslave actions into dafilesrv with new operations.
When copying from/to a bare metal system the requests are sent to the dafilesrv for the node that currently runs ftslave. For a container system the requests are sent to the loadbalanced service.
It might be possible to migrate to lamda style functions for some of the work...
A later optimization would use a cloud service where it was possible.
When local split points are supported it may be better to spray a file 1:1 along with partition information. Even without local split points it may still be better to spray a file 1:1 (cheaper).
What are the spray targets? It may need to be storage plane + number of parts, rather than a target cluster. The default number of parts is the #devices on the storage plane.
=> Milestones a) Move ftslave code to dafilesrv (partition, pull, push) [Should be included in 7.12.x stream to allow remote read compatibility?] b) Create a dafilesrv component to the helm charts, with internal and external services. c) use storage planes to determine how files are sprayed etc. (bare-metal, #devices) Adapt dfu/fileservices calls to take (storageplane,number) instead of cluster. There should already be a 1:1 mapping from existing cluster to storage planes in a bare-metal system, so this may not involve much work. [May also need a flag to indicate if ._1_of_1 is appended?] d) Select correct dafilesrv for bare-metal storage planes, or load balanced service for other. (May need to think through how remote files are represented.)
=> Can import from a bare metal system or a containerized system using command line??
: NOTE: Bare-metal to containerized will likely need push operations on the bare-metal system. (And therefore serialized security information) This may still cause issues since it is unlikely containerized will be able to pull from bare-metal. Pushing, but not creating a logical file entry on the containerized system should be easier since it can use a local storage plane definition.
e) Switch over to using the esp based meta information, so that it can include details of storage planes and secrets.
: [Note this would also need to be in 7.12.x to allow remote export to containerized, that may well be a step too far]
f) Add option to configure the number of file parts for spray/copy/despray g) Ensure that eclwatch picks up the list of storage planes (and the default number of file parts), and has ability to specify #parts.
Later: h) plan how cloud-services can be used for some of the copies i) investigate using serverless functions to calculate split points. j) Use refactored disk read/write interfaces to clean up read and copy code. k) we may not want to expose access keys to allow remote reads/writes - in which they would need to be pushed from a bare-metal dafilesrv to a containerized dafilesrv.
Other dependencies: * Refactored file meta information. If this is switching to being plane based, then the meta information should also be plane based. Main difference is not including the path in the meta information (can just be ignored) * esp service for getting file information. When reading remotely it needs to go via this now...
`,80)]))}const m=t(r,[["render",s]]);export{f as __pageData,m as default};
diff --git a/assets/devdoc_README.md.D15419X_.js b/assets/devdoc_README.md.D15419X_.js
new file mode 100644
index 00000000000..41f34472bd1
--- /dev/null
+++ b/assets/devdoc_README.md.D15419X_.js
@@ -0,0 +1 @@
+import{_ as t,c as a,a3 as i,o}from"./chunks/framework.DkhCEVKm.js";const c=JSON.parse('{"title":"Developer Documentation","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/README.md","filePath":"devdoc/README.md","lastUpdated":1731340314000}'),r={name:"devdoc/README.md"};function n(l,e,s,d,m,u){return o(),a("div",null,e[0]||(e[0]=[i('
The ECL language is documented in the ecl language reference manual (generated as ECLLanguageReference-<version>.pdf).
',9)]))}const f=t(r,[["render",n]]);export{c as __pageData,f as default};
diff --git a/assets/devdoc_README.md.D15419X_.lean.js b/assets/devdoc_README.md.D15419X_.lean.js
new file mode 100644
index 00000000000..41f34472bd1
--- /dev/null
+++ b/assets/devdoc_README.md.D15419X_.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as a,a3 as i,o}from"./chunks/framework.DkhCEVKm.js";const c=JSON.parse('{"title":"Developer Documentation","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/README.md","filePath":"devdoc/README.md","lastUpdated":1731340314000}'),r={name:"devdoc/README.md"};function n(l,e,s,d,m,u){return o(),a("div",null,e[0]||(e[0]=[i('
The ECL language is documented in the ecl language reference manual (generated as ECLLanguageReference-<version>.pdf).
',9)]))}const f=t(r,[["render",n]]);export{c as __pageData,f as default};
diff --git a/assets/devdoc_SecurityConfig.md.C2-pRa18.js b/assets/devdoc_SecurityConfig.md.C2-pRa18.js
new file mode 100644
index 00000000000..3dbc504c3d4
--- /dev/null
+++ b/assets/devdoc_SecurityConfig.md.C2-pRa18.js
@@ -0,0 +1 @@
+import{_ as t,c as r,a3 as a,o}from"./chunks/framework.DkhCEVKm.js";const h=JSON.parse('{"title":"Security Configuration","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/SecurityConfig.md","filePath":"devdoc/SecurityConfig.md","lastUpdated":1731340314000}'),n={name:"devdoc/SecurityConfig.md"};function d(s,e,i,u,c,l){return o(),r("div",null,e[0]||(e[0]=[a('
This document covers security configuration values and meanings. It does not serve as the source for how to configure security, but rather what the different values mean. These are not covered in the docs nor does any reasonable help information exist in the config manager or yaml files.
Security is configured either through an LDAP server or a plugin. Additionally, these are supported in both legacy deployments that use environment.xml and containerized deployments using Kubernetes and Helm charts. While these methods differ, the configuration values remain the same. Focus is placed on the different values and not the deployment target. Differences based on deployment can be found in the relevant platform documents.
Security is implemented via a security manager interface. Managers are loaded and used by components within the system to check authorization and authentication. LDAP is an exception to the loadable manager model. It is not a compliant loadable module like other security plugins. For that reason, the configuration for each is separated into two sections below: LDAP and Plugin Security Managers.
LDAP is a protocol that connects to an Active Directory server (AD). The term LDAP is used interchangeably with AD. Below are the configuration values for an LDAP connection. These are valid for both legacy (environment.xml) and containerized deployments. For legacy deployments the configuration manager is the primary vehicle for setting these values. However, some values are not available through the tool and must be set manually in the environment.xml if needed for a legacy deployment.
In containerized environments, a LDAP configuration block is required for each component. Currently, this results in a verbose configuration where much of the information is repeated.
LDAP is capable if handling user authentication and feature access authorization (such as filescopes).
Value
Example
Meaning
adminGroupName
HPCCAdmins
Group name containing admin users for the AD
cacheTimeout
60
Timeout in minutes to keep cached security data
ldapCipherSuite
N/A
Used when AD is not up to date with latest SSL libs. AD admin must provide
ldapPort
389 (default)
Insecure port
ldapSecurePort
636 (default)
Secure port over TLS
ldapProtocol
ldap
ldap for insecure (default), using ldapPort ldaps for secure using ldapSecurePort
ldapTimeoutSec
60 (default 5 for debug, 60 otherwise)
Connection timeout to an AD before rollint to next AD
Appears to only be used for IPlanet type ADs, but may still be required
systemCommonName
hpccAdmin
AD username of user to proxy all AD operations
systemPassword
System user password
AD user password
ldapAdminSecretKey
none
Key for Kubernetes secrets (4) (5)
ldapAdminVaultId
none
Vault ID used to load system username and password (5)
ldapDomain
none
Appears to be a comma separated version of the AD domain name components (5)
ldapAddress
192.168.10.42
IP address to the AD
commonBasedn
DC=z0lpf,DC=onmicrosoft,DC=com
Overrides the domain retrieved from the AD for the system user (5)
templateName
none
Template used when adding resources (5)
authMethod
none
Not sure yet
Notes:
modulesBaseDn is the same as resourcesBaseDn The code looks for first for modulesBaseDn and if not found will search for resourcesBaseDn
Allowed values for serverType are ActiveDirectory, AzureActiveDirectory, 389DirectoryServer, OpenLDAP, Fedora389
For AzureAD, users are managed from the AD dashboard, not via ECLWatch or through LDAP
If present, ldapAdminVaultId is read and systemCommonName and systemPassword are read from the Kubernetes secrets store and not from the LDAP config values
Must be configured manually in the environment.xml in legacy environments
Plugin security managers are separate shared objects loaded and initialized by the system. The manager interface is passed to components in order to provide necessary security functions. Each plugin has its own configuration. HPCC components can be configured to use a plugin as needed.
',21)]))}const p=t(n,[["render",d]]);export{h as __pageData,p as default};
diff --git a/assets/devdoc_SecurityConfig.md.C2-pRa18.lean.js b/assets/devdoc_SecurityConfig.md.C2-pRa18.lean.js
new file mode 100644
index 00000000000..3dbc504c3d4
--- /dev/null
+++ b/assets/devdoc_SecurityConfig.md.C2-pRa18.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as r,a3 as a,o}from"./chunks/framework.DkhCEVKm.js";const h=JSON.parse('{"title":"Security Configuration","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/SecurityConfig.md","filePath":"devdoc/SecurityConfig.md","lastUpdated":1731340314000}'),n={name:"devdoc/SecurityConfig.md"};function d(s,e,i,u,c,l){return o(),r("div",null,e[0]||(e[0]=[a('
This document covers security configuration values and meanings. It does not serve as the source for how to configure security, but rather what the different values mean. These are not covered in the docs nor does any reasonable help information exist in the config manager or yaml files.
Security is configured either through an LDAP server or a plugin. Additionally, these are supported in both legacy deployments that use environment.xml and containerized deployments using Kubernetes and Helm charts. While these methods differ, the configuration values remain the same. Focus is placed on the different values and not the deployment target. Differences based on deployment can be found in the relevant platform documents.
Security is implemented via a security manager interface. Managers are loaded and used by components within the system to check authorization and authentication. LDAP is an exception to the loadable manager model. It is not a compliant loadable module like other security plugins. For that reason, the configuration for each is separated into two sections below: LDAP and Plugin Security Managers.
LDAP is a protocol that connects to an Active Directory server (AD). The term LDAP is used interchangeably with AD. Below are the configuration values for an LDAP connection. These are valid for both legacy (environment.xml) and containerized deployments. For legacy deployments the configuration manager is the primary vehicle for setting these values. However, some values are not available through the tool and must be set manually in the environment.xml if needed for a legacy deployment.
In containerized environments, a LDAP configuration block is required for each component. Currently, this results in a verbose configuration where much of the information is repeated.
LDAP is capable if handling user authentication and feature access authorization (such as filescopes).
Value
Example
Meaning
adminGroupName
HPCCAdmins
Group name containing admin users for the AD
cacheTimeout
60
Timeout in minutes to keep cached security data
ldapCipherSuite
N/A
Used when AD is not up to date with latest SSL libs. AD admin must provide
ldapPort
389 (default)
Insecure port
ldapSecurePort
636 (default)
Secure port over TLS
ldapProtocol
ldap
ldap for insecure (default), using ldapPort ldaps for secure using ldapSecurePort
ldapTimeoutSec
60 (default 5 for debug, 60 otherwise)
Connection timeout to an AD before rollint to next AD
Appears to only be used for IPlanet type ADs, but may still be required
systemCommonName
hpccAdmin
AD username of user to proxy all AD operations
systemPassword
System user password
AD user password
ldapAdminSecretKey
none
Key for Kubernetes secrets (4) (5)
ldapAdminVaultId
none
Vault ID used to load system username and password (5)
ldapDomain
none
Appears to be a comma separated version of the AD domain name components (5)
ldapAddress
192.168.10.42
IP address to the AD
commonBasedn
DC=z0lpf,DC=onmicrosoft,DC=com
Overrides the domain retrieved from the AD for the system user (5)
templateName
none
Template used when adding resources (5)
authMethod
none
Not sure yet
Notes:
modulesBaseDn is the same as resourcesBaseDn The code looks for first for modulesBaseDn and if not found will search for resourcesBaseDn
Allowed values for serverType are ActiveDirectory, AzureActiveDirectory, 389DirectoryServer, OpenLDAP, Fedora389
For AzureAD, users are managed from the AD dashboard, not via ECLWatch or through LDAP
If present, ldapAdminVaultId is read and systemCommonName and systemPassword are read from the Kubernetes secrets store and not from the LDAP config values
Must be configured manually in the environment.xml in legacy environments
Plugin security managers are separate shared objects loaded and initialized by the system. The manager interface is passed to components in order to provide necessary security functions. Each plugin has its own configuration. HPCC components can be configured to use a plugin as needed.
',21)]))}const p=t(n,[["render",d]]);export{h as __pageData,p as default};
diff --git a/assets/devdoc_SecurityUserAuthentication.md.DbBHZTGf.js b/assets/devdoc_SecurityUserAuthentication.md.DbBHZTGf.js
new file mode 100644
index 00000000000..30f445cdd01
--- /dev/null
+++ b/assets/devdoc_SecurityUserAuthentication.md.DbBHZTGf.js
@@ -0,0 +1 @@
+import{_ as t,c as a,a3 as i,o as s}from"./chunks/framework.DkhCEVKm.js";const p=JSON.parse('{"title":"User Authentication","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/SecurityUserAuthentication.md","filePath":"devdoc/SecurityUserAuthentication.md","lastUpdated":1731340314000}'),r={name:"devdoc/SecurityUserAuthentication.md"};function n(o,e,h,c,u,d){return s(),a("div",null,e[0]||(e[0]=[i('
This document covers user authentication, the process of verifying the identity of the user. Authorization is a separate topic covering whether a user should be allowed to perform a specific operation of access a specific resource.
Each supported security manager is covered.
Generally, when authentication is needed, the security manager client should call the ISecManagerauthenticateUser method. The method also allows the caller to detect if the user being authenticated is a superuser. Use of that feature is beyond the scope of this document. In practice, this method is rarely if ever called. User authentication is generally performed as part of authorization. This is covered in more detail below.
This section covers how each supported security manager handles user authentication. As stated above, the method authenticateUser is defined for this purpose. However, other methods also perform user authentication. The sections that follow describe in general how each security manager performs user authentication, whether from directly calling the authenticateUser method, or as an ancillary action taken when another method is called.
The LDAP security manager uses the configured Active Directory to authenticate users. Once authenticated, the user is added to the permissions cache, if enabled, to prevent repeated trips to the AD whenever an authentication check is required.
If caching is enabled, a lookup is done to see if the user is already cached. If so, the cached user authentication status is returned. Note that the cached status remains until either the cache time to live expires or is cleared either manually or through some other programmatic action.
If caching is not enabled, a request is sent to the AD to validate the user credentials.
In either case, if digital signatures are configured, the user is also digitally signed using the username. Digitally signing the user allows for quick authentication by validating the signature against the username. During initial authentication, if the digital signature exists, it is verified to provide a fast way to authenticate the user. If the signature is not verified, the user is marked as not authenticated.
Authentication status is stored in the security user object so that further checks are not necessary when the same user object is used in multiple calls to the security manager.
Authentication in the htpasswd manager does not support singularly authenticating the user without also authorizing resource access. See the special case for authentication with authorization below.
Regardless, the htpasswd manager authenticates users using the .htpasswd file that is installed on the cluster. It does so by finding the user in the file and verifying that the input hashed password matches the stored hashed password in file.
The single user security manager allows the definition of a single username with a password. The values are set in the environment configuration and are read during the initialization of the manager. All authentication requests validate against the configured username and password. The process is a simple comparison. Note that the password stored in the environment is hashed.
Since resource access authorization requires an authenticated user, the authorization process also authenticates the user before checking authorization. There are a couple of advantages to this
Two separate calls are not required to check authorization (one to verify the user and one to check authorization)
The caller can perform third party authorization for specific user access
The authenticate method, or any of its overloads or derivatives, accepts a resource or resource list and a user. These methods authenticate the user first before checking access to the specified resource.
ECL Watch uses user authentication during authorization during its log in process. Instead of first authenticating the user, it calls an authenticate method passing both the user and the necessary resources for which the user must have access in order to log into ECL Watch.
',22)]))}const m=t(r,[["render",n]]);export{p as __pageData,m as default};
diff --git a/assets/devdoc_SecurityUserAuthentication.md.DbBHZTGf.lean.js b/assets/devdoc_SecurityUserAuthentication.md.DbBHZTGf.lean.js
new file mode 100644
index 00000000000..30f445cdd01
--- /dev/null
+++ b/assets/devdoc_SecurityUserAuthentication.md.DbBHZTGf.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as a,a3 as i,o as s}from"./chunks/framework.DkhCEVKm.js";const p=JSON.parse('{"title":"User Authentication","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/SecurityUserAuthentication.md","filePath":"devdoc/SecurityUserAuthentication.md","lastUpdated":1731340314000}'),r={name:"devdoc/SecurityUserAuthentication.md"};function n(o,e,h,c,u,d){return s(),a("div",null,e[0]||(e[0]=[i('
This document covers user authentication, the process of verifying the identity of the user. Authorization is a separate topic covering whether a user should be allowed to perform a specific operation of access a specific resource.
Each supported security manager is covered.
Generally, when authentication is needed, the security manager client should call the ISecManagerauthenticateUser method. The method also allows the caller to detect if the user being authenticated is a superuser. Use of that feature is beyond the scope of this document. In practice, this method is rarely if ever called. User authentication is generally performed as part of authorization. This is covered in more detail below.
This section covers how each supported security manager handles user authentication. As stated above, the method authenticateUser is defined for this purpose. However, other methods also perform user authentication. The sections that follow describe in general how each security manager performs user authentication, whether from directly calling the authenticateUser method, or as an ancillary action taken when another method is called.
The LDAP security manager uses the configured Active Directory to authenticate users. Once authenticated, the user is added to the permissions cache, if enabled, to prevent repeated trips to the AD whenever an authentication check is required.
If caching is enabled, a lookup is done to see if the user is already cached. If so, the cached user authentication status is returned. Note that the cached status remains until either the cache time to live expires or is cleared either manually or through some other programmatic action.
If caching is not enabled, a request is sent to the AD to validate the user credentials.
In either case, if digital signatures are configured, the user is also digitally signed using the username. Digitally signing the user allows for quick authentication by validating the signature against the username. During initial authentication, if the digital signature exists, it is verified to provide a fast way to authenticate the user. If the signature is not verified, the user is marked as not authenticated.
Authentication status is stored in the security user object so that further checks are not necessary when the same user object is used in multiple calls to the security manager.
Authentication in the htpasswd manager does not support singularly authenticating the user without also authorizing resource access. See the special case for authentication with authorization below.
Regardless, the htpasswd manager authenticates users using the .htpasswd file that is installed on the cluster. It does so by finding the user in the file and verifying that the input hashed password matches the stored hashed password in file.
The single user security manager allows the definition of a single username with a password. The values are set in the environment configuration and are read during the initialization of the manager. All authentication requests validate against the configured username and password. The process is a simple comparison. Note that the password stored in the environment is hashed.
Since resource access authorization requires an authenticated user, the authorization process also authenticates the user before checking authorization. There are a couple of advantages to this
Two separate calls are not required to check authorization (one to verify the user and one to check authorization)
The caller can perform third party authorization for specific user access
The authenticate method, or any of its overloads or derivatives, accepts a resource or resource list and a user. These methods authenticate the user first before checking access to the specified resource.
ECL Watch uses user authentication during authorization during its log in process. Instead of first authenticating the user, it calls an authenticate method passing both the user and the necessary resources for which the user must have access in order to log into ECL Watch.
',22)]))}const m=t(r,[["render",n]]);export{p as __pageData,m as default};
diff --git a/assets/devdoc_StyleGuide.md.6RHWelMX.js b/assets/devdoc_StyleGuide.md.6RHWelMX.js
new file mode 100644
index 00000000000..16271d18871
--- /dev/null
+++ b/assets/devdoc_StyleGuide.md.6RHWelMX.js
@@ -0,0 +1,51 @@
+import{_ as e,c as i,a3 as a,o as t}from"./chunks/framework.DkhCEVKm.js";const k=JSON.parse('{"title":"Coding conventions","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/StyleGuide.md","filePath":"devdoc/StyleGuide.md","lastUpdated":1731340314000}'),n={name:"devdoc/StyleGuide.md"};function l(o,s,h,r,p,d){return t(),i("div",null,s[0]||(s[0]=[a(`
Everyone has their own ideas of what the best code formatting style is, but most would agree that code in a mixture of styles is the worst of all worlds. A consistent coding style makes unfamiliar code easier to understand and navigate.
In an ideal world, the HPCC sources would adhere to the coding standards described perfectly. In reality, there are many places that do not. These are being cleaned up as and when we find time.
Unlike most software projects around, HPCC has some very specific constraints that makes most basic design decisions difficult, and often the results are odd to developers getting acquainted with its code base. For example, when HPCC was initially developed, most common-place libraries we have today (like STL and Boost) weren't available or stable enough at the time.
Also, at the beginning, both C++ and Java were being considered as the language of choice, but development started with C++. So a C++ library that copied most behaviour of the Java standard library (At the time, Java 1.4) was created (see jlib below) to make the transition, if ever taken, easier. The transition never happened, but the decisions were taken and the whole platform is designed on those terms.
Most importantly, the performance constraints in HPCC can make no-brainer decisions look impossible in HPCC. One example is the use of traditional smart pointers implementations (such as boost::shared_ptr or C++'s auto_ptr), that can lead to up to 20% performance hit if used instead of our internal shared pointer implementation.
The last important point to consider is that some libraries/systems were designed to replace older ones but haven't got replaced yet. There is a slow movement to deprecate old systems in favour of consolidating a few ones as the elected official ways to use HPCC (Thor, Roxie) but old systems still could be used for years in tests or legacy sub-systems.
In a nutshell, expect re-implementation of well-known containers and algorithms, expect duplicated functionality of sub-systems and expect to be required to use less-friendly libraries for the sake of performance, stability and longevity.
We use the extension .cpp for C++ source files, and .h or .hpp for header files. Header files with the .hpp extension should be used for headers that are internal to a single library, while header files with the .h extension should be used for the interface that the library exposes. There will typically be one .h file per library, and one .hpp file per cpp file.
Source file names within a single shared library should share a common prefix to aid in identifying where they belong.
Header files with extension .ipp (i for internal) and .tpp (t for template) will be phased out in favour of the scheme described above.
We adopted a Java-like inheritance model, with macro substitution for the basic Java keywords. This changes nothing on the code, but make it clearer for the reader on what's the recipient of the inheritance doing with it's base.
interface (struct): declares an interface (pure virtual class)
extends (public): One interface extending another, both are pure virtual
implements (public): Concrete class implementing an interface
There is no semantic check, which makes it difficult to enforce such scheme, which has led to code not using it intermixed with code using it. You should use it when possible, most importantly on code that already uses it.
We also tend to write methods inline, which matches well with C++ Templates requirements. We, however, do not enforce the one-class-per-file rule.
See the Interfaces section for more information on our implementation of interfaces.
Class and interface names are in CamelCase with a leading capital letter. Interface names should be prefixed capital I followed by another capital. Class names may be prefixed with a C if there is a corresponding I-prefixed interface name, e.g. when the interface is primarily used to create an opaque type, but need not be otherwise.
Variables, function and method names, and parameters use camelCase starting with a lower case letter. Parameters may be prefixed with underscore when the parameter is used to initialize a member variable of the same name. Common cases are constructors and setter methods.
Use real pointers when you can, and smart pointers when you have to. Take extra care on understanding the needs of your pointers and their scope. Most programs can afford a few dangling pointers, but a high-performance clustering platform cannot.
Most importantly, use common sense and a lot of thought. Here are a few guidelines:
Use real pointers for return values, parameter passing.
For .md variables use real pointers if their lifetime is guaranteed to be longer than the function (and no exception is thrown from functions you call), shared pointers otherwise.
Use Shared pointers for member variables - unless there is a strong guarantee the object has a longer lifetime.
Create Shared<X> with either:
Owned<X>: if your new pointer will take ownership of the pointer
Linked<X>: if you are sharing the ownership (shared)
Warning: Direct manipulation of the ownership might cause Shared<> pointers to lose the pointers, so subsequent calls to it (like o2->doIt() after o3 gets ownership) will cause segmentation faults.
Refer to [Reference counted objects]{.title-ref} for more information on our smart pointer implementation, Shared<>.
Methods that return pointers to link counted objects, or that use them, should use a common naming standard:
Foo * queryFoo() Does not return a linked pointer since lifetime is guaranteed for a set period. Caller should link if it needs to retain it for longer.
Foo * getFoo() Returned value is linked and should be assigned to an owned, or returned directly.
void setFoo(Foo * x) Generally parameters to functions are assumed to be owned by the caller, the callee needs to link them if they are retained.
void setFoo(Foo * ownedX) Some calls do transfer ownership of parameters - the parameter should be named to indicate this. If the function only has a single signficant parameter then sometimes the name of the function indicates the ownership.
We use 4 spaces to indent each level. TAB characters should not be used.
The { that starts a new scope and the corresponding } to close it are placed on a new line by themselves, and are not indented. This is sometimes known as the Allman or ANSI style.
We generally believe in the philosophy that well written code is self-documenting. Comments are also encouraged to describe why something is done, rather than how - which should be clear from the code.
javadoc-formatted comments for classes and interfaces are being added.
The virtual keyword should be included on the declaration of all virtual functions - including those in derived classes, and the override keyword should be used on all virtual functions in derived classes.
While C++ does not have explicit support for interfaces (in the java sense), an abstract class with no data members and all functions pure virtual can be used in the same way.
Interfaces are pure virtual classes. They are similar concepts to Java's interfaces and should be used on public APIs. If you need common code, use policies (see below).
An interface's name must start with an 'I' and the base class for its concrete implementations should start with a 'C' and have the same name, ex:
cpp
CFoo : implements IFoo { };
When an interface has multiple implementations, try to stay as close as possible to this rule. Ex:
Or, for partial implementation, use something like this:
cpp
CFoo : implements IFoo { };
+CFooCool : public CFoo { };
+CFooWarm : public CFoo { };
Extend current interfaces only on a 'is-a' approach, not to aggregate functionality. Avoid pollution of public interfaces by having only the public methods on the most-base interface in the header, and internal implementation in the source file. Prefer pImpl idiom (pointer-to-implementation) for functionality-only requirements and policy based design for interface requirements.
Example 1: You want to decouple part of the implementation from your class, and this part does not implements the interface your contract requires.:
cpp
interface IFoo
+{
+ virtual void foo()=0;
+};
+
+// Following is implemented in a separate private file...
+class CFoo : implements IFoo
+{
+ MyImpl *pImpl;
+public:
+ virtual void foo() override { pImpl->doSomething(); }
+};
Example2: You want to implement the common part of one (or more) interface(s) in a range of sub-classes.:
Shared<> is an in-house intrusive smart pointer implementation. It is close to boost's intrusive_ptr. It has two derived implementations: Linked and Owned, which are used to control whether the pointer is linked when a shared pointer is created from a real pointer or not, respectively. Ex:
cpp
Owned<Foo> myFoo = new Foo; // Take owenership of the pointers
+Linked<Foo> anotherFoo = = myFoo; // Shared ownership
Shared<> is thread-safe and uses atomic reference count handled by each object (rather than by the smart pointer itself, like boost's shared_ptr).
This means that, to use Shared<>, your class must implement the Link() and Release() methods - most commonly by extending the CInterfaceOf<> class, or the CInterface class (and using the IMPLEMENT_IINTERFACE macro in the public section of your class declaration).
This interface controls how you Link() and Release() the pointer. This is necessary because in some inner parts of HPCC, the use of a "really smart" smart pointer would add too many links and releases (on temporaries, local variables, members, etc) that could add to a significant performance hit.
The CInterface implementation also include a virtual function beforeDispose() which is called before the object is deleted. This allows resources to be cleanly freed up, with the full class hierarchy (including virtual functions) available even when freeing items in base classes. It is often used for caches that do not cause the objects to be retained.
Requiring more work: * namespaces * STL * c++11 * Review all documentation * Better examples for shared
`,84)]))}const g=e(n,[["render",l]]);export{k as __pageData,g as default};
diff --git a/assets/devdoc_StyleGuide.md.6RHWelMX.lean.js b/assets/devdoc_StyleGuide.md.6RHWelMX.lean.js
new file mode 100644
index 00000000000..16271d18871
--- /dev/null
+++ b/assets/devdoc_StyleGuide.md.6RHWelMX.lean.js
@@ -0,0 +1,51 @@
+import{_ as e,c as i,a3 as a,o as t}from"./chunks/framework.DkhCEVKm.js";const k=JSON.parse('{"title":"Coding conventions","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/StyleGuide.md","filePath":"devdoc/StyleGuide.md","lastUpdated":1731340314000}'),n={name:"devdoc/StyleGuide.md"};function l(o,s,h,r,p,d){return t(),i("div",null,s[0]||(s[0]=[a(`
Everyone has their own ideas of what the best code formatting style is, but most would agree that code in a mixture of styles is the worst of all worlds. A consistent coding style makes unfamiliar code easier to understand and navigate.
In an ideal world, the HPCC sources would adhere to the coding standards described perfectly. In reality, there are many places that do not. These are being cleaned up as and when we find time.
Unlike most software projects around, HPCC has some very specific constraints that makes most basic design decisions difficult, and often the results are odd to developers getting acquainted with its code base. For example, when HPCC was initially developed, most common-place libraries we have today (like STL and Boost) weren't available or stable enough at the time.
Also, at the beginning, both C++ and Java were being considered as the language of choice, but development started with C++. So a C++ library that copied most behaviour of the Java standard library (At the time, Java 1.4) was created (see jlib below) to make the transition, if ever taken, easier. The transition never happened, but the decisions were taken and the whole platform is designed on those terms.
Most importantly, the performance constraints in HPCC can make no-brainer decisions look impossible in HPCC. One example is the use of traditional smart pointers implementations (such as boost::shared_ptr or C++'s auto_ptr), that can lead to up to 20% performance hit if used instead of our internal shared pointer implementation.
The last important point to consider is that some libraries/systems were designed to replace older ones but haven't got replaced yet. There is a slow movement to deprecate old systems in favour of consolidating a few ones as the elected official ways to use HPCC (Thor, Roxie) but old systems still could be used for years in tests or legacy sub-systems.
In a nutshell, expect re-implementation of well-known containers and algorithms, expect duplicated functionality of sub-systems and expect to be required to use less-friendly libraries for the sake of performance, stability and longevity.
We use the extension .cpp for C++ source files, and .h or .hpp for header files. Header files with the .hpp extension should be used for headers that are internal to a single library, while header files with the .h extension should be used for the interface that the library exposes. There will typically be one .h file per library, and one .hpp file per cpp file.
Source file names within a single shared library should share a common prefix to aid in identifying where they belong.
Header files with extension .ipp (i for internal) and .tpp (t for template) will be phased out in favour of the scheme described above.
We adopted a Java-like inheritance model, with macro substitution for the basic Java keywords. This changes nothing on the code, but make it clearer for the reader on what's the recipient of the inheritance doing with it's base.
interface (struct): declares an interface (pure virtual class)
extends (public): One interface extending another, both are pure virtual
implements (public): Concrete class implementing an interface
There is no semantic check, which makes it difficult to enforce such scheme, which has led to code not using it intermixed with code using it. You should use it when possible, most importantly on code that already uses it.
We also tend to write methods inline, which matches well with C++ Templates requirements. We, however, do not enforce the one-class-per-file rule.
See the Interfaces section for more information on our implementation of interfaces.
Class and interface names are in CamelCase with a leading capital letter. Interface names should be prefixed capital I followed by another capital. Class names may be prefixed with a C if there is a corresponding I-prefixed interface name, e.g. when the interface is primarily used to create an opaque type, but need not be otherwise.
Variables, function and method names, and parameters use camelCase starting with a lower case letter. Parameters may be prefixed with underscore when the parameter is used to initialize a member variable of the same name. Common cases are constructors and setter methods.
Use real pointers when you can, and smart pointers when you have to. Take extra care on understanding the needs of your pointers and their scope. Most programs can afford a few dangling pointers, but a high-performance clustering platform cannot.
Most importantly, use common sense and a lot of thought. Here are a few guidelines:
Use real pointers for return values, parameter passing.
For .md variables use real pointers if their lifetime is guaranteed to be longer than the function (and no exception is thrown from functions you call), shared pointers otherwise.
Use Shared pointers for member variables - unless there is a strong guarantee the object has a longer lifetime.
Create Shared<X> with either:
Owned<X>: if your new pointer will take ownership of the pointer
Linked<X>: if you are sharing the ownership (shared)
Warning: Direct manipulation of the ownership might cause Shared<> pointers to lose the pointers, so subsequent calls to it (like o2->doIt() after o3 gets ownership) will cause segmentation faults.
Refer to [Reference counted objects]{.title-ref} for more information on our smart pointer implementation, Shared<>.
Methods that return pointers to link counted objects, or that use them, should use a common naming standard:
Foo * queryFoo() Does not return a linked pointer since lifetime is guaranteed for a set period. Caller should link if it needs to retain it for longer.
Foo * getFoo() Returned value is linked and should be assigned to an owned, or returned directly.
void setFoo(Foo * x) Generally parameters to functions are assumed to be owned by the caller, the callee needs to link them if they are retained.
void setFoo(Foo * ownedX) Some calls do transfer ownership of parameters - the parameter should be named to indicate this. If the function only has a single signficant parameter then sometimes the name of the function indicates the ownership.
We use 4 spaces to indent each level. TAB characters should not be used.
The { that starts a new scope and the corresponding } to close it are placed on a new line by themselves, and are not indented. This is sometimes known as the Allman or ANSI style.
We generally believe in the philosophy that well written code is self-documenting. Comments are also encouraged to describe why something is done, rather than how - which should be clear from the code.
javadoc-formatted comments for classes and interfaces are being added.
The virtual keyword should be included on the declaration of all virtual functions - including those in derived classes, and the override keyword should be used on all virtual functions in derived classes.
While C++ does not have explicit support for interfaces (in the java sense), an abstract class with no data members and all functions pure virtual can be used in the same way.
Interfaces are pure virtual classes. They are similar concepts to Java's interfaces and should be used on public APIs. If you need common code, use policies (see below).
An interface's name must start with an 'I' and the base class for its concrete implementations should start with a 'C' and have the same name, ex:
cpp
CFoo : implements IFoo { };
When an interface has multiple implementations, try to stay as close as possible to this rule. Ex:
Or, for partial implementation, use something like this:
cpp
CFoo : implements IFoo { };
+CFooCool : public CFoo { };
+CFooWarm : public CFoo { };
Extend current interfaces only on a 'is-a' approach, not to aggregate functionality. Avoid pollution of public interfaces by having only the public methods on the most-base interface in the header, and internal implementation in the source file. Prefer pImpl idiom (pointer-to-implementation) for functionality-only requirements and policy based design for interface requirements.
Example 1: You want to decouple part of the implementation from your class, and this part does not implements the interface your contract requires.:
cpp
interface IFoo
+{
+ virtual void foo()=0;
+};
+
+// Following is implemented in a separate private file...
+class CFoo : implements IFoo
+{
+ MyImpl *pImpl;
+public:
+ virtual void foo() override { pImpl->doSomething(); }
+};
Example2: You want to implement the common part of one (or more) interface(s) in a range of sub-classes.:
Shared<> is an in-house intrusive smart pointer implementation. It is close to boost's intrusive_ptr. It has two derived implementations: Linked and Owned, which are used to control whether the pointer is linked when a shared pointer is created from a real pointer or not, respectively. Ex:
cpp
Owned<Foo> myFoo = new Foo; // Take owenership of the pointers
+Linked<Foo> anotherFoo = = myFoo; // Shared ownership
Shared<> is thread-safe and uses atomic reference count handled by each object (rather than by the smart pointer itself, like boost's shared_ptr).
This means that, to use Shared<>, your class must implement the Link() and Release() methods - most commonly by extending the CInterfaceOf<> class, or the CInterface class (and using the IMPLEMENT_IINTERFACE macro in the public section of your class declaration).
This interface controls how you Link() and Release() the pointer. This is necessary because in some inner parts of HPCC, the use of a "really smart" smart pointer would add too many links and releases (on temporaries, local variables, members, etc) that could add to a significant performance hit.
The CInterface implementation also include a virtual function beforeDispose() which is called before the object is deleted. This allows resources to be cleanly freed up, with the full class hierarchy (including virtual functions) available even when freeing items in base classes. It is often used for caches that do not cause the objects to be retained.
Requiring more work: * namespaces * STL * c++11 * Review all documentation * Better examples for shared
`,84)]))}const g=e(n,[["render",l]]);export{k as __pageData,g as default};
diff --git a/assets/devdoc_UserBuildAssets.md.RFQFpFTM.js b/assets/devdoc_UserBuildAssets.md.RFQFpFTM.js
new file mode 100644
index 00000000000..2501d958ea6
--- /dev/null
+++ b/assets/devdoc_UserBuildAssets.md.RFQFpFTM.js
@@ -0,0 +1,34 @@
+import{_ as s,c as a,a3 as t,o as n}from"./chunks/framework.DkhCEVKm.js";const o="/HPCC-Platform/assets/repository-tag-tab.DSfut5r1.png",i="/HPCC-Platform/assets/actions-secrets-and-variables.BfAcH5xn.png",r="/HPCC-Platform/assets/HPCC-12345-build-in-progress.DU0-PikH.png",f=JSON.parse('{"title":"Build Assets for individual developer","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/UserBuildAssets.md","filePath":"devdoc/UserBuildAssets.md","lastUpdated":1731340314000}'),l={name:"devdoc/UserBuildAssets.md"};function p(c,e,d,u,h,g){return n(),a("div",null,e[0]||(e[0]=[t('
The modern tool used for generating all our official assets is the Github Actions build-asset workflow on the hpcc-systems/HPCC-Platform repository, located here. Developers and contributors can utilize this same workflow on their own forked repository. This allows developers to quickly create assets for testing changes and test for errors before the peer review process.
Build assets will generate every available project under the HPCC-Platform namespace. There currently is not an option to control which packages in the build matrix get generated. But most packages get built in parallel, and released after the individual matrix job is completed, so there is no waiting on packages you don't need. Exceptions to this are for packages that require other builds to complete, such as the ECLIDE.
Upon completion of each step and matrix job in the workflow, the assets will be output to the repositories tags tab. An example for the hpcc-systems user repository is hpcc-systems/HPCC-Platform/tags.
The build assets workflow requires several repository secrets be available on a developers machine in order to run properly. You can access these secrets and variables by going to the settings tab in your forked repository, and then clicking on the Secrets and Variables - Actions drop down under Security on the lefthand side of the settings screen.
Create a secret by clicking the green New Repository Secret button. The following secrets are needed;
LNB_ACTOR - Your Github username
LNB_TOKEN - Classic Github token for your user with LN repo access
DOCKER_USERNAME - Your docker.io username
DOCKER_PASSWORD - Your docker.io password
SIGNING_CERTIFICATE - pks12 self signed cert encoded to base64 for windows signing
SIGNING_CERTIFICATE_PASSPHRASE - passphrase for pks12 cert
SIGNING_SECRET - ssh-keygen private key for signing linux builds
SIGN_MODULES_KEYID - email used to generate key
SIGN_MODULES_PASSPHRASE - passphrase for private key
The build-asset workflow is kicked off by a tag being pushed to the developers HPCC-Platform repository. Before we push the tag to our HPCC-Platform repository, we will want to have other tags in place if we want LN and ECLIDE builds to function correctly. Suggested tag patterns are community_HPCC-12345-rc1 or HPCC-12345-rc1.
If you choose not to tag the LN and ECLIDE builds, the community builds will generate but errors will be thrown for any build utilizing the LN repository. ECLIDE will not even attempt a build unless you are also successfully building LN due to the dependency scheme we use. The 'Baremetal' builds are designed to generate our clienttools targets for windows-2022 and macos-12 distributions. These jobs contain both the COMMUNITY and LN builds. If the LN build is not tagged, the COMMUNITY section of the job will run, and the assets will be uploaded, but the job will fail when it tries to build LN.
If you choose to precede your Jira number with community_ then you must tag LN with internal_ and ECLIDE with eclide_. Otherwise just use the Jira tag in all three repositories.
Once the LN and ECLIDE repository tags have been created and pushed with the same base branch that your work is based on for the HPCC-Platform, then you are free to push the HPCC-Platform tag which will initiate the build process.
The summary of the build-asset workflow can then be viewed for progress, and individual jobs can be selected to check build outputs.
Assets from the workflow will be released into the corresponding tag location, either in the HPCC-Platform repository for all community based builds, or the LN repository for any builds containing proprietary plugins. Simply browse to the releases or tag tab of your repository and select the tag name you just built. The assets will show up there as the build completes. An example of this on the hpcc-systems repository is hpcc-systems/HPCC-Platform/releases.
',54)]))}const m=s(l,[["render",p]]);export{f as __pageData,m as default};
diff --git a/assets/devdoc_UserBuildAssets.md.RFQFpFTM.lean.js b/assets/devdoc_UserBuildAssets.md.RFQFpFTM.lean.js
new file mode 100644
index 00000000000..2501d958ea6
--- /dev/null
+++ b/assets/devdoc_UserBuildAssets.md.RFQFpFTM.lean.js
@@ -0,0 +1,34 @@
+import{_ as s,c as a,a3 as t,o as n}from"./chunks/framework.DkhCEVKm.js";const o="/HPCC-Platform/assets/repository-tag-tab.DSfut5r1.png",i="/HPCC-Platform/assets/actions-secrets-and-variables.BfAcH5xn.png",r="/HPCC-Platform/assets/HPCC-12345-build-in-progress.DU0-PikH.png",f=JSON.parse('{"title":"Build Assets for individual developer","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/UserBuildAssets.md","filePath":"devdoc/UserBuildAssets.md","lastUpdated":1731340314000}'),l={name:"devdoc/UserBuildAssets.md"};function p(c,e,d,u,h,g){return n(),a("div",null,e[0]||(e[0]=[t('
The modern tool used for generating all our official assets is the Github Actions build-asset workflow on the hpcc-systems/HPCC-Platform repository, located here. Developers and contributors can utilize this same workflow on their own forked repository. This allows developers to quickly create assets for testing changes and test for errors before the peer review process.
Build assets will generate every available project under the HPCC-Platform namespace. There currently is not an option to control which packages in the build matrix get generated. But most packages get built in parallel, and released after the individual matrix job is completed, so there is no waiting on packages you don't need. Exceptions to this are for packages that require other builds to complete, such as the ECLIDE.
Upon completion of each step and matrix job in the workflow, the assets will be output to the repositories tags tab. An example for the hpcc-systems user repository is hpcc-systems/HPCC-Platform/tags.
The build assets workflow requires several repository secrets be available on a developers machine in order to run properly. You can access these secrets and variables by going to the settings tab in your forked repository, and then clicking on the Secrets and Variables - Actions drop down under Security on the lefthand side of the settings screen.
Create a secret by clicking the green New Repository Secret button. The following secrets are needed;
LNB_ACTOR - Your Github username
LNB_TOKEN - Classic Github token for your user with LN repo access
DOCKER_USERNAME - Your docker.io username
DOCKER_PASSWORD - Your docker.io password
SIGNING_CERTIFICATE - pks12 self signed cert encoded to base64 for windows signing
SIGNING_CERTIFICATE_PASSPHRASE - passphrase for pks12 cert
SIGNING_SECRET - ssh-keygen private key for signing linux builds
SIGN_MODULES_KEYID - email used to generate key
SIGN_MODULES_PASSPHRASE - passphrase for private key
The build-asset workflow is kicked off by a tag being pushed to the developers HPCC-Platform repository. Before we push the tag to our HPCC-Platform repository, we will want to have other tags in place if we want LN and ECLIDE builds to function correctly. Suggested tag patterns are community_HPCC-12345-rc1 or HPCC-12345-rc1.
If you choose not to tag the LN and ECLIDE builds, the community builds will generate but errors will be thrown for any build utilizing the LN repository. ECLIDE will not even attempt a build unless you are also successfully building LN due to the dependency scheme we use. The 'Baremetal' builds are designed to generate our clienttools targets for windows-2022 and macos-12 distributions. These jobs contain both the COMMUNITY and LN builds. If the LN build is not tagged, the COMMUNITY section of the job will run, and the assets will be uploaded, but the job will fail when it tries to build LN.
If you choose to precede your Jira number with community_ then you must tag LN with internal_ and ECLIDE with eclide_. Otherwise just use the Jira tag in all three repositories.
Once the LN and ECLIDE repository tags have been created and pushed with the same base branch that your work is based on for the HPCC-Platform, then you are free to push the HPCC-Platform tag which will initiate the build process.
The summary of the build-asset workflow can then be viewed for progress, and individual jobs can be selected to check build outputs.
Assets from the workflow will be released into the corresponding tag location, either in the HPCC-Platform repository for all community based builds, or the LN repository for any builds containing proprietary plugins. Simply browse to the releases or tag tab of your repository and select the tag name you just built. The assets will show up there as the build completes. An example of this on the hpcc-systems repository is hpcc-systems/HPCC-Platform/releases.
',54)]))}const m=s(l,[["render",p]]);export{f as __pageData,m as default};
diff --git a/assets/devdoc_VersionSupport.md.DqUAdEd9.js b/assets/devdoc_VersionSupport.md.DqUAdEd9.js
new file mode 100644
index 00000000000..3bd74af2ecb
--- /dev/null
+++ b/assets/devdoc_VersionSupport.md.DqUAdEd9.js
@@ -0,0 +1 @@
+import{_ as t,c as i,a3 as a,o as s}from"./chunks/framework.DkhCEVKm.js";const p=JSON.parse('{"title":"Current versions","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/VersionSupport.md","filePath":"devdoc/VersionSupport.md","lastUpdated":1731340314000}'),o={name:"devdoc/VersionSupport.md"};function r(n,e,l,h,c,u){return s(),i("div",null,e[0]||(e[0]=[a('
We release a new version of the platform every 3 months. If there are major changes in functionality, or significant backward compatibility issues then it will be tagged as a new major version, otherwise a new minor version. We normally maintain 4 versions of the system, which means that each new release will typically be supported for a year. Once a new major or minor version has been tagged gold it should not have any changes that change the behavior of queries.
Which versions should changes be applied to? The following gives some examples of the types of changes and which version they would be most relevant to target.
"master":
New features.
Bug fixes that will change the semantics of existing queries or processes.
Refactoring.
Performance improvements (unless simple and safe)
"<current>":
Bug fixes that only change behavior where it previously crashes or had undefined behavior (If well defined but wrong need to have very strong justification to change.)
Fixes for race conditions (the behavior was previously indeterminate so less of an argument against it changing)
Data corruption fixes - on a case by case basis if they change existing query results.
Missing functionality that prevents features from working.
Changes for tech-preview work that only effect those who are using it.
Regressions.
Improvements to logging and error messages (possibly in "previous" if simple and added to help diagnose problems).
Occasional simple refactoring that makes up-merging simpler..
Changes to improve backward compatibility of new features. (E.g. adding an ignored syntax to the compiler.)
Performance improvements - if simple and safe
"<previous>":
Simple bug fixes that do not change behavior
Simple changes for missing functionality
Regressions with simple fixes (but care is needed if it caused a change in behavior)
Serious regressions
Complex security fixes
"<critical>" fixes only:
Simple security fixes
Complex security fixes if sufficiently serious
"<security>" fixes only:
Serious security fixes
Occasionally earlier branches will be chosen, (e.g. security fixes to even older versions) but they should always be carefully discussed (and documented).
We aim to produce new point releases once a week. The point releases will contain
a) Any changes to the code base for that branch. b) Any security fixes for libraries that are project dependencies. We will upgrade to the latest point release for the library that fixes the security issue. c) For the cloud any security fixes in the base image or the packages installed in that image.
If there are no changes in any of those areas for a particular version then a new point release will not be created.
If you are deploying a system to the cloud you have one of two options
a) Use the images that are automatically built and published as part of the build pipeline. This image is currently based on ubuntu 22.04 and contains the majority of packages users will require.
b) Use your own hardened base image, and install the containerized package that we publish into that image.
We currently generate the following versions of the package and images:
debug
release with symbols
release without symbols.
It is recommended that you deploy the "release with symbols" version to all bare-metal and non-production cloud deployments. The extra symbols allow the system to generate stack backtraces which make it much easier to diagnose problems if they occur. The "release without symbols" version is recommended for Kubernetes production deployments. Deploying a system without symbols reduces the size of the images. This reduces the time it takes Kubernetes to copy the image before provisioning a new node.
',27)]))}const m=t(o,[["render",r]]);export{p as __pageData,m as default};
diff --git a/assets/devdoc_VersionSupport.md.DqUAdEd9.lean.js b/assets/devdoc_VersionSupport.md.DqUAdEd9.lean.js
new file mode 100644
index 00000000000..3bd74af2ecb
--- /dev/null
+++ b/assets/devdoc_VersionSupport.md.DqUAdEd9.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as i,a3 as a,o as s}from"./chunks/framework.DkhCEVKm.js";const p=JSON.parse('{"title":"Current versions","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/VersionSupport.md","filePath":"devdoc/VersionSupport.md","lastUpdated":1731340314000}'),o={name:"devdoc/VersionSupport.md"};function r(n,e,l,h,c,u){return s(),i("div",null,e[0]||(e[0]=[a('
We release a new version of the platform every 3 months. If there are major changes in functionality, or significant backward compatibility issues then it will be tagged as a new major version, otherwise a new minor version. We normally maintain 4 versions of the system, which means that each new release will typically be supported for a year. Once a new major or minor version has been tagged gold it should not have any changes that change the behavior of queries.
Which versions should changes be applied to? The following gives some examples of the types of changes and which version they would be most relevant to target.
"master":
New features.
Bug fixes that will change the semantics of existing queries or processes.
Refactoring.
Performance improvements (unless simple and safe)
"<current>":
Bug fixes that only change behavior where it previously crashes or had undefined behavior (If well defined but wrong need to have very strong justification to change.)
Fixes for race conditions (the behavior was previously indeterminate so less of an argument against it changing)
Data corruption fixes - on a case by case basis if they change existing query results.
Missing functionality that prevents features from working.
Changes for tech-preview work that only effect those who are using it.
Regressions.
Improvements to logging and error messages (possibly in "previous" if simple and added to help diagnose problems).
Occasional simple refactoring that makes up-merging simpler..
Changes to improve backward compatibility of new features. (E.g. adding an ignored syntax to the compiler.)
Performance improvements - if simple and safe
"<previous>":
Simple bug fixes that do not change behavior
Simple changes for missing functionality
Regressions with simple fixes (but care is needed if it caused a change in behavior)
Serious regressions
Complex security fixes
"<critical>" fixes only:
Simple security fixes
Complex security fixes if sufficiently serious
"<security>" fixes only:
Serious security fixes
Occasionally earlier branches will be chosen, (e.g. security fixes to even older versions) but they should always be carefully discussed (and documented).
We aim to produce new point releases once a week. The point releases will contain
a) Any changes to the code base for that branch. b) Any security fixes for libraries that are project dependencies. We will upgrade to the latest point release for the library that fixes the security issue. c) For the cloud any security fixes in the base image or the packages installed in that image.
If there are no changes in any of those areas for a particular version then a new point release will not be created.
If you are deploying a system to the cloud you have one of two options
a) Use the images that are automatically built and published as part of the build pipeline. This image is currently based on ubuntu 22.04 and contains the majority of packages users will require.
b) Use your own hardened base image, and install the containerized package that we publish into that image.
We currently generate the following versions of the package and images:
debug
release with symbols
release without symbols.
It is recommended that you deploy the "release with symbols" version to all bare-metal and non-production cloud deployments. The extra symbols allow the system to generate stack backtraces which make it much easier to diagnose problems if they occur. The "release without symbols" version is recommended for Kubernetes production deployments. Deploying a system without symbols reduces the size of the images. This reduces the time it takes Kubernetes to copy the image before provisioning a new node.
',27)]))}const m=t(o,[["render",r]]);export{p as __pageData,m as default};
diff --git a/assets/devdoc_Workunits.md.DJg1TI-D.js b/assets/devdoc_Workunits.md.DJg1TI-D.js
new file mode 100644
index 00000000000..b025fe5667f
--- /dev/null
+++ b/assets/devdoc_Workunits.md.DJg1TI-D.js
@@ -0,0 +1,403 @@
+import{_ as i,c as a,a3 as t,o as n}from"./chunks/framework.DkhCEVKm.js";const g=JSON.parse('{"title":"Understanding workunits","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/Workunits.md","filePath":"devdoc/Workunits.md","lastUpdated":1731340314000}'),h={name:"devdoc/Workunits.md"};function e(l,s,k,p,r,E){return n(),a("div",null,s[0]||(s[0]=[t(`
A workunit contains all the information that the system requires about a query - including the parameters it takes, how to execute it, and how to format the results. Understanding the contents of a workunit is a key step to understanding how the HPCC system fits together. This document begins with an overview of the different elements in a workunit. That is then followed by a walk-through executing a simple query, with a more detailed description of some of the workunit components to show how they all tie together.
Before looking at the contents of a workunit it is important to understand one of the design goals behind the HPCC system. The HPCC system logically splits the code that is needed to execute a query in two. On the one hand there are the algorithms that are used to perform different dataset operations (e.g. sorting, deduping). The same algorithms are used by all the queries that execute on the system. On the other hand there is the meta-data that describes the columns present in the datasets, which columns you need to sort by, and the order of operations required by the query. These are typically different for each query. This "meta-data" includes generated compare functions, classes that describe the record formats, serialized data and graphs.
A workunit only contains data and code relating to the meta data for the query, i.e. "what to do", while the different engines (hthor, roxie, and thor) implement the algorithms - "how to do it". If you look at a workunit for an ECL query that sorts a dataset you will not find code to perform the sort itself in the workunit - you will not even find a call to a sort library function - that logic is contained in the engine that executes the query.
One consequence of this split, which can be initially confusing, is that execution continually passes back and forth between the generated code and the engines. By the end of this document you should have a better understanding of how the generated code is structured and the part it plays in executing a query.
Note the term "Query" is used as a generic term to cover read-only queries (typically run in roxie) and ETL (Extract, Transform, and Load) style queries that create lots of persistent datafiles (typically run in Thor). Also, the term "workunit" is used ambiguously. The dll created from a query is called a workunit (which is static), but "workunit" is also used to describe a single execution of a query (which includes the parameters and results). It should be clear from the context which of these is meant.
Throughout this document "dll" is a generic term used to refer to a dynamically loaded library. These correspond to shared objects in Linux (extension '.so'), dynamic libraries in Max OS X ('.dylib'), and dynamic link libraries in windows ('.dll').
A workunit is generated by the ecl compiler, and consists of a single dll. That dll contains several different elements:
Various C++ helper classes, and exported factory methods are used to create instances of those classes.
An XML resource containing different pieces of information about a workunit, including workflow and graphs.
Other user-defined resources included in the manifest.
A workunit dll contains everything that the engines need to execute the query. When a workunit is executed, key elements of the xml information are cloned from the workunit dll and copied into a database. This is then augmented with other information as the query is executed - e.g. input parameters, results, statistics, etc.. The contents of the workunit are accessed through an "IWorkUnit" interface (defined in common/workunit/workunit.hpp) that hides the implementation details.
(Workunit information is currently stored in the Dali database - one of the components within the HPCC platform. Work is in-progress to allow the bulk of this workunit data to be stored in Cassandra or another third-party database instead.)
The workunit information is used by most of the components in the system. The following is a quick outline:
eclcc
Creates a workunit dll from an ecl query.
eclccserver
Executes eclcc to create a workunit dll, and then clones some of the information into dali to create an active instance, ready to execute.
esp
Uses information in the workunit dll to publish workunits. This includes details of the parameters that the query takes, how they should be formatted, and the results it returns.
eclscheduler
Monitors workunits that are waiting for events, and updates them when those events occur.
eclagent/Roxie
Process the different workflow actions, and workflow code.
hThor/Roxie/Thor
Execute graphs within the workflow items.
Dali
This database is used to store the state of the workunit state.
The following ECL will be used as an example for the rest of the discussion. It is a very simple search that takes a string parameter 'searchName', which is the name of the person to search for, and returns the matching records from an index. It also outputs the word 'Done!' as a separate result.
This section outlines the different sections in a workunit. This is followed by a walk-through of the stages that take place when a workunit is executed, together with a more detailed explanation of the workunit contents.
The workflow is the highest level of control within a workunit. It is used for two related purposes:
Scheduling The HPCC system allows ECL code to be executed when certain events occur - e.g. every hour or when files are uploaded to a landing zone. Each piece of ECL code that is triggered by an external event creates a separate workflow action. This allows each of those events to be processed independently.
Splitting up queries. There are situations where it is useful to break parts of an ECL query into independent sections. The simplest example is the PERSIST workflow operation, which allows results to be shared between different work units. Each workflow operation creates one (or sometimes more) independent workflow items, which are then connected together.
Each piece of independent ECL is given a unique workflow id (wfid). Often workflow items need to be executed in a particular order, e.g. ensuring a persist exists before using it, which is managed with dependencies between different workflow items.
Our example above generates the following XML entry in the workunit:
This contains two workflow items. The first workflow item (wfid=1) ensures that the stored value has a default value if it has not been supplied. The second item (with wfid=2) is the main code for the query. This has a dependency on the first workflow item because the stored variable needs to be intialised before it is executed.
The generated code contains a class instance that is used for executing the code associated with the workflow items. It is generated at the end of the main C++ module. E.g.:
cpp
struct MyEclProcess : public EclProcess {
+ virtual int perform(IGlobalCodeContext * gctx, unsigned wfid) {
+ ....
+ switch (wfid) {
+ case 1U:
+ ... code for workflow item 1 ...
+ case 2U:
+ ... code for workflow item 2 ...
+ break;
+ }
+ return 2U;
+ }
+};
The main element is a switch statement inside the perform() function that allows the workflow engines to execute the code associated with a particular workflow item.
There is also an associated factory function that is exported from the dll, and is used by the engines to create instances of the class:
Most of the work executing a query involves processing dataset operations, which are implemented as a graph of activities. Each graph is represented in the workunit as an xml graph structure (currently it uses the xgmml format). The graph xml includes details of which types of activities are required to be executed, how they are linked together, and any other dependencies.
This graph contains a single subgraph (node id=2) that contains two activities - an index read activity and an output result activity. These activities are linked by a single edge (id "2_0"). The details of the contents are covered in the section on executing graphs below.
Each activity has a corresponding class instance in the generated code, and a factory function for creating instances of that class:
cpp
struct cAc2 : public CThorIndexReadArg {
+ ... Implementation of the helper for activity #2 ...
+};
+extern "C" ECL_API IHThorArg * fAc2() { return new cAc2; }
+
+struct cAc3 : public CThorWorkUnitWriteArg {
+ ... Implementation of the helper for activity #3 ...
+};
+extern "C" ECL_API IHThorArg * fAc3() { return new cAc3; }
The helper class for an activity implements the interface that is required for that particular kind. (The interfaces are defined in rtl/include/eclhelper.hpp - further details below.)
The are several other items, detailed below, that are logically associated with a workunit. The information may be stored in the workunit dll or in various other location e.g. Dali, Sasha or Cassandra. It is all accessed through the IWorkUnit interface in common/workunit/workunit.hpp that hides the implementation details. For instance information generated at runtime cannot by definition be included in the workunit dll.
Many queries contain input parameters that modify their behaviour. These correspond to STORED definitions in the ECL. Our example contains a single string "searchName", so the workunit contains a single input parameter:
in our example there are two - the dataset of results and the text string "Done!". The values of the results for a query are associated with the workunit. (They are currently saved in dali, but this may change in version 6.0.)
Other statistics and timings created when running the query are stored in the runtime copy of the workunit. (Statistics for graph elements are stored in a different format from global statistics, but the IWorkUnit interface ensures the implementation details are hidden.)
It is possible to include other user-defined resources in the workunit dll - e.g. web pages, or dashboard layouts. I have to confess I do not understand them... ??Tony please provide some more information....!
Once a workunit has been compiled to a dll it is ready to be executed. Execution can be triggered in different ways, E.g.:
The ECL for a query is submitted to esp
A workunit entry, containing the ECL, is created in dali and added to an eclccserver queue.
An eclccserver instance removes the workunit form the queue, and compiles the ECL to a workunit dll.
The dali workunit entry is updated with the information from the workunit dll.
The dali workunit is added to the agent execution queue associated with the target cluster.
The associated engine (actually agentexec for hThor and Thor) pulls a query form the queue and executes it.
A query is submitted and published with a name. Another request is then submitted to execute this previously compiled query.
A workunit entry, containing the ECL, is created in dali and added to an eclccserver queue.
An eclccserver instance removes the workunit form the queue, and compiles the ECL to a workunit dll.
There is a 'query set' for each combination of query name and the target cluster. The new workunit dll is added to the appropriate query set, and marked as the current active implementation.
Later, a query that references a named query is submitted to esp.
The name and target cluster are mapped via the query set to the active implementation, and a workunit instance is created from the active workunit dll.
The workunit is added to a roxie or eclagentexec queue ready to be executed.
The associated engine pulls a query form the queue and executes it.
A query is compiled as a stand alone executable. The executable is then run.
eclcc is executed on the command line without the -shared command line option.
The resulting executable is run. The engine used to execute the query depends on the -platform parameter supplied to eclcc.
Most queries create persistent workunits in dali and then update those workunits with results as they are calculated, however for some roxie queries (e.g. in a production system) the execution workunits are transient.
The following walk-through details the main stages executing a query, and the effect each of the query elements has.
The system uses several inter-process queues to communicate between the different components in the system. These queues are implemented by dali. Components can subscribe to one or more queues, and receive notifications when entries are avaialable.
Some example queues are:
<cluster>.eclserver - workunits to be compiled
<cluster>.roxie - workunits to execute on roxie
<cluster>.thor - graphs to execute on thor
<cluster>.eclscheduler - workunits that need to wait for events
<cluster>.agent - workunits to be executed on hthor or thor.
dfuserver_queue - dfu workunits for sprays/file copies etc.
When a workunit is ready to be run, the workflow controls the flow of execution. The workflow engine (defined in common/workunit/workflow.cpp) is responsible for determining which workflow item should be executed next.
The workflow for Thor and hThor jobs is coordinated by eclagent, while roxie includes the workflow engine in its process. The eclscheduler also uses the workflow engine to process events and mark workflow items ready for execution.
eclagent, or roxie calls the createProcess() function from the workunit dll to create an instance of the generated workflow helper class, and passes it to the workflow engine. The workflow engine walks the workflow items to find any items that are ready to be executed (have the state "reqd" - i.e. required). If a required workflow item has dependencies on other child workflow items then those children are executed first. Once all dependencies have executed successfully the parent workflow item is executed. The example has the following workflow entries:
Item 2 has a state of "reqd", so it should be evaluated now. Item 2 has a dependency on item 1, so that must be evaluated first. This is achieved by calling MyEclProcess::perform() on the object that was previously created from the workunit dll, passing in wfid = 1. That will execute the following code:
cpp
switch (wfid) {
+ case 1U:
+ if (!gctx->isResult("searchname",4294967295U)) {
+ ctx->setResultString("searchname",4294967295U,5U,"Smith");
+ }
+ break;
+ break;
+}
This checks if a value has been provided for the input parameter, and if not assigns a default value of "Smith". The function returns control to the workflow engine. With the dependencies for wfid 2 now satisfied, the generated code for that workflow id is now executed:
Most of the work for this workflow item involves executing graph1 (by calling back into eclagent/roxie). However, the code also directly sets another result. This is fairly typical - the code inside MyProcess::perform often combines evaluating scalar results, executing graphs, and calling functions that cannot (currently) be called inside a graph (e.g. those involving superfile transactions).
Once all of the required workflow items are executed, the workunit is marked as completed. Alternatively, if there are workflow items that are waiting to be triggered by an event, the workunit will be passed to the scheduler, which will keep monitoring for events.
Note that most items in the xml workflow begin in the state WFStateNull. This means that it is valid to execute them, but they haven't been executed yet. Typically, only a few items begin with the state WFStateReqd.
There are various specialised types of workflow items - e.g. sequential, wait, independent, but they all follow the same basic approach of executing dependencies and then executing that particular item.
Most of the interesting work in an ECL query is done within a graph. The call ctx->executeGraph will either execute the graph locally (in the case of hthor and roxie), or add the workunit onto a queue (for Thor). Whichever happens, control will pass to that engine.
Each item mode/type can affect the dependency structure of the workflow:
Sequential/Ordered
The workflow structure for sequential and ordered is the same. An item is made to contain all of the actions in the statement. This is achieved by making each action a dependency of this item. The dependencies, and consequently the actions, must be executed in order. An extra item is always inserted before each dependency. This means that if other statements reference the same dependency, it will only be performed once.
Persist
When the persist workflow service is used, two items are created. One item contains the graphs that perform the expression defined in ECL. It also stores the wfid for the second item. The second item is used to determine whether the persist is up to date.
Condition (IF)
The IF function has either 2 or 3 arguments: the expression, the trueresult, and sometimes the falseresult. For each argument, a workflow item is created. These items are stored as dependencies to the condition, in the order stated above.
Contingency (SUCCESS/FAILURE)
When a contingency clause is defined for an attribute, the attribute item stores the wfid of the contingency. If both success and failure are used, then the item will store the wfid of each contingency. The contingency is composed of items, just like the larger query.
Recovery
When a workflow item fails, if it has a recovery clause, the item will be re-executed. The clause contains actions defined by the programmer to remedy the problem. This clause is stored differently to SUCCESS/FAILURE, in that the recovery clause is a dependency of the failed item. In order to stop the recovery clause from being executed like the other dependencies, it is marked with WFStateSkip.
Independent
This specifier is used when a programmer wants to common up code for the query. It prevents the same piece of code from being executed twice in different contexts. To achieve this, an extra item is inserted between the expression and whichever items depend on it. This means that although the attribute can be referenced several times, it will only be executed once.
All the engines (roxie, hThor, Thor) execute graphs in a very similar way. The main differences are that hThor and Thor execute a sub graph at a time, while roxie executes a complete graph as one. Roxie is also optimized to minimize the overhead of executing a query - since the same query tends to be run multiple times. This means that roxie creates a graph of factory objects and those are then used to create the activities. The core details are the same for each of them though.
Each graph (e.g. graph1) consists of 1 or more subgraphs (in the example above, node id=1). Each of those subgraphs contains 1 or more activities (node id=2, node id=3).
The xml for each activity might contain the following information:
A unique id (e.g. id="2").
The "kind" of the activity, e.g. <att name="_kind" value="77"/>. The value is an item from the enum ThorActivityKind in eclhelper.hpp.
The ECL that created the activity. E.g. <att name="ecl" value="...">
The identifier of the ecl definition. E.g. <att name="name" value="results"/>
Location (e.g. file, line number, column) of the original ECL. E.g. <att name="definition" value="workuniteg1.ecl(3,1)"/>
Meta information the code generator has deduced about the activity output. Examples include the record size, expected number of rows, sort order etc. E.g. <att name="recordSize" value="120"/>
Hints, which are used for fine control of options for a particular activity (e.g,, the number of threads to use while sorting).
Record counts and stats once the job has executed. (These are logically associated with the activities in the graph, but stored separately.)
Graphs also contain edges that can take one of 3 forms:
Edges within graphs
: These are used to indicate how the activities are connected. The source activity is used as the input to the target activity. These edges have the following format:
<edge id="<source-activity-id>_<output-count>" source="<source-activity-id>" target="<target-activity-id">
+
+There is only one edge in our example workunit: \\<edge id="2\\_0"
+source="2" target="3"/\\>.
+
Edges between graphs
: These are used to indicate direct dependencies between activities. For instance there will be an edge connecting the activity that writes a spill file to the activity that reads it. These edges have the following format:
<edge id="<source-activity-id>_<target-activity-id>" source="<source-subgraph-id>" target="<target-subgraph-id>"
+ <att name="_sourceActivity" value="<source-activity-id>"/>
+ <att name="_targetActivity" value="<target-activity-id>"/>
+ </edge>
+
+Roxie often optimizes spilled datasets and treats these edges as
+equivalent to the edges between activities.
+
Other dependencies.
: These are similar to the edges between graphs, but they are used for values that are used within an activity. For instance one part of the graph may calculate the maximum value of a column, and another activity may filter out all records that do not match that maximum value. The format is the same as the edges between graphs except that the edge contains the following attribute:
<att name="_dependsOn" value="1"/>
+
Each activity in a graph also has a corresponding helper class instance in the generated code. (The name of the class is "cAc" followed by the activity number, and the exported factory method is "fAc" followed by the activity number.) Each helper class implements a specialised interface (derived from IHThorArg) - the particular interface is determined by the value of the "_kind" attribute for the activity.
The contents of file rtl/include/eclhelper.hpp is key to understanding how the generated code relates to the activities. Each kind of activity requires a helper class that implements a specific interface. The helpers allow the engine to tailor the generalised activity implementation to the the particular instance - e.g. for a filter activity whether a row should be included or excluded. The appendix at the end of this document contains some further information about this file.
The classes in the generated workunits are normally derived from base implementations of those interfaces (which are implemented in rtl/include/eclhelper_base.hpp). This reduces the size of the generated code by providing default implementations for various functions.
For instance the helper for the index read (activity 2) is defined as:
a) onCreate() - common to all activities. It is called by the engine when the helper is first created, and allows the helper to cache information that does not change - in this case the name that is being searched for. b) getFileName() - determines the name of the index being read. c) createSegmentMonitors() - defines which columns to filter, and which values to match against. d) transform() - create the record to return from the activity. It controls which columns should be included from the index row in the output. (In this case all.)
To execute a graph, the engine walks the activities in the graph xml and creates, in memory, a graph of implementation activities.
For each activity, the name of the helper factory is calculated from the activity number (e.g. fAc2 for activity 2). That function is imported from the loaded dll, and then called to create an instance of the generated helper class - in this case cAc2.
The engine then creates an instance of the class for implementing the activity, and passes the previously created helper object to the constructor. The engine uses the _kind attribute in the graph to determine which activity class should be used. E.g. In the example above activity 2 has a _kind of 77, which corresponds to TAKindexread. For an index-read activity roxie will create an instance of CRoxieServerIndexReadActivity. (The generated helper that is passed to the activity instance will implement IHThorIndexReadArg). The activity implementations may also extract other information from the xml for the activity - e.g. hints. Once all the activities are created the edge information is used to link inputs activities to output activities and add other dependencies.
Note: Any subgraph that is marked with <att name="rootGraph" value="1"/> is a root subgraph. An activity within a subgraph that has no outputs is called a 'sink' (and an activity without any inputs is called a 'source').
Executing a graph involves executing all the root subgraphs that it contains. All dependencies of the activities within the subgraph must be executed before a subgraph is executed. To execute a subgraph, the engine executes each of the sink activities on separate threads, and then waits for each of those threads to complete. Each sink activity lazily pulls input rows on demand from activities further up the graph, processes them and returns when complete.
(If you examine the code you will find that this is a simplification. The implementation for processing dependencies is more fine grained to ensure IF datasets, OUPUT(,UPDATE) and other ECL constructs are executed correctly.)
In our example the execution flows as follows:
Only a single root subgraph, so need to execute that.
The engine will execute activity 3 - the workunit-write activity (TAKworkunitwrite).
The workunit-write activity will start its input.
The index-read activity will call the helper functions to obtain the filename, resolve the index, and create the filter.
The workunit-write activity requests a row from its input.
The index-read finds the first row, and calls the helper's transform() method to create an output row.
The workunit-write activity persists the row to a buffer (using the serializer provided by the IOutputMetaData interface implemented by the class mx1).
Back to step 5, workunit-write reading a row from its input, until end of file is returned (notified as two consecutive NULL rows.
Workunit-write commits the results and finishes.
The execution generally switches back and forth between the code in the engines, and the members of the generated helper classes.
There are some other details of query execution that are worth highlighting:
Child Queries
: Some activities perform complicated operations on child datasets of the input rows. E.g. remove all duplicate people who are marked as living at this address. This will create a "child query" in the graph - i.e. a nested graph within a subgraph, which may be executed each time a new input row is processed by the containing activity. (The graph of activities for each child query is created at the same time as the parent activity. The activity instances are reinitialised and re-executed for each input row processed by the parent activity to minimise the create-time overhead.)
Other helper functions
: The generated code contains other functions that are used to describe the meta information for the rows processed within the graph. E.g. the following class describes the output from the index read activity:
This represents a fixed size row that occupies 120 bytes. The object
+returned by the queryTypeInfo() function provides information about
+the types of the fields:
+
I.e. a string column, length 40 called "name", followed by a
+string column, length 80 called "address". The interface
+IOutputMetaData in eclhelper.hpp is key to understanding how the
+rows are processed.
+
Inline dataset operations
: The rule mentioned at the start - that the generated code does not contain any knowledge of how to perform a particular dataset operation - does have one notable exception. Some operations on child datasets are very simple to implement, and more efficient if they are implemented using inline C++ code. (The generated code is smaller, and they avoid the overhead of setting up a child graph.) Examples include filtering and aggregating column values from a child dataset.
The full code in the different engines is more complicated than the simplified process outlined above, especially when it comes to executing dependencies, but the broad outline is the same.
More information on the work done in the code generator to create the workunit can be found in ecl/eclcc/DOCUMENTATION.rst.
The C++ code can be generated as a single C++ file or multiple files. The system defaults to multiple C++ files, so that they can be compiled in parallel (and to avoid problems some compilers have with very large files). When multipe C++ files are generated the metadata classes and workflow classes are generated in the main module, and the activities are generated in the files suffixed with a number. It may be easier to understand the generated code if it is in one place. In which case, compile your query with the option -fspanMultipleCpp=0. Use -fsaveCppTempFiles to ensure the C++ files are not deleted (the C++ files will appear as helpers in the workunit details).
: The interface that is used by the workflow engine to execute the different workflow items in the generated code.
ThorActivityKind
: This enumeration contains one entry for each activity supported by the engines.
ICodeContext
: This interface is implemented by the engine, and provides a mechanism for the generated code to call back into the engine. For example resolveChildQuery() is used to obtain a reference to a child query that can then be executed later.
IOutputMetaData
: This interface is used to describe any meta data associated with the data being processed by the queries.
IHThorArg
: The base interface for defining information about an activity. Each activity has an associated interface that is derived from this interface. E.g. each instance of the sort activity will have a helper class implementing IHThorSortArg in the generated query. There is normally a corresponding base class for each interface in eclhelper_base.hpp that is used by the generated code e.g. CThorSortArg.
ARowBuilder
: This abstract base class is used by the transform functions to reserve memory for the rows that are created.
IEngineRowAllocator
: Used by the generated code to allocate rows and rowsets. Can also be used to release rows (or call the global function rtlReleaseRow()).
IGlobalCodeContext
: Provides access to functions that cannot be called inside a graph - i.e. can only be called from the global workflow code. Most functions are related to the internal implementation of particular workflow item types (e.g. persists).
: An activity is the basic unit of dataset processing implemented by the engines. Each activity corresponds to a node in the thor execution graph. Instances of the activities are connnected together to create the graph.
dll
: A dynamically loaded library. These correspond to shared objects in Linux (extension '.so'), dynamic libraries in Max OS X ('.dylib'), and dynamic link libraries in windows ('.dll').
superfile
: A composite file which allows a collection of files to be treated as a single compound file.
The XML for a workunit can be viewed on the XML tag in eclwatch, or generated by compiling the ECL using the -wu option with eclcc. Alternatively eclcc -b -S can be used to generate the XML and the C++ at the same time (the output filenames are derived from the input name).
`,170)]))}const o=i(h,[["render",e]]);export{g as __pageData,o as default};
diff --git a/assets/devdoc_Workunits.md.DJg1TI-D.lean.js b/assets/devdoc_Workunits.md.DJg1TI-D.lean.js
new file mode 100644
index 00000000000..b025fe5667f
--- /dev/null
+++ b/assets/devdoc_Workunits.md.DJg1TI-D.lean.js
@@ -0,0 +1,403 @@
+import{_ as i,c as a,a3 as t,o as n}from"./chunks/framework.DkhCEVKm.js";const g=JSON.parse('{"title":"Understanding workunits","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/Workunits.md","filePath":"devdoc/Workunits.md","lastUpdated":1731340314000}'),h={name:"devdoc/Workunits.md"};function e(l,s,k,p,r,E){return n(),a("div",null,s[0]||(s[0]=[t(`
A workunit contains all the information that the system requires about a query - including the parameters it takes, how to execute it, and how to format the results. Understanding the contents of a workunit is a key step to understanding how the HPCC system fits together. This document begins with an overview of the different elements in a workunit. That is then followed by a walk-through executing a simple query, with a more detailed description of some of the workunit components to show how they all tie together.
Before looking at the contents of a workunit it is important to understand one of the design goals behind the HPCC system. The HPCC system logically splits the code that is needed to execute a query in two. On the one hand there are the algorithms that are used to perform different dataset operations (e.g. sorting, deduping). The same algorithms are used by all the queries that execute on the system. On the other hand there is the meta-data that describes the columns present in the datasets, which columns you need to sort by, and the order of operations required by the query. These are typically different for each query. This "meta-data" includes generated compare functions, classes that describe the record formats, serialized data and graphs.
A workunit only contains data and code relating to the meta data for the query, i.e. "what to do", while the different engines (hthor, roxie, and thor) implement the algorithms - "how to do it". If you look at a workunit for an ECL query that sorts a dataset you will not find code to perform the sort itself in the workunit - you will not even find a call to a sort library function - that logic is contained in the engine that executes the query.
One consequence of this split, which can be initially confusing, is that execution continually passes back and forth between the generated code and the engines. By the end of this document you should have a better understanding of how the generated code is structured and the part it plays in executing a query.
Note the term "Query" is used as a generic term to cover read-only queries (typically run in roxie) and ETL (Extract, Transform, and Load) style queries that create lots of persistent datafiles (typically run in Thor). Also, the term "workunit" is used ambiguously. The dll created from a query is called a workunit (which is static), but "workunit" is also used to describe a single execution of a query (which includes the parameters and results). It should be clear from the context which of these is meant.
Throughout this document "dll" is a generic term used to refer to a dynamically loaded library. These correspond to shared objects in Linux (extension '.so'), dynamic libraries in Max OS X ('.dylib'), and dynamic link libraries in windows ('.dll').
A workunit is generated by the ecl compiler, and consists of a single dll. That dll contains several different elements:
Various C++ helper classes, and exported factory methods are used to create instances of those classes.
An XML resource containing different pieces of information about a workunit, including workflow and graphs.
Other user-defined resources included in the manifest.
A workunit dll contains everything that the engines need to execute the query. When a workunit is executed, key elements of the xml information are cloned from the workunit dll and copied into a database. This is then augmented with other information as the query is executed - e.g. input parameters, results, statistics, etc.. The contents of the workunit are accessed through an "IWorkUnit" interface (defined in common/workunit/workunit.hpp) that hides the implementation details.
(Workunit information is currently stored in the Dali database - one of the components within the HPCC platform. Work is in-progress to allow the bulk of this workunit data to be stored in Cassandra or another third-party database instead.)
The workunit information is used by most of the components in the system. The following is a quick outline:
eclcc
Creates a workunit dll from an ecl query.
eclccserver
Executes eclcc to create a workunit dll, and then clones some of the information into dali to create an active instance, ready to execute.
esp
Uses information in the workunit dll to publish workunits. This includes details of the parameters that the query takes, how they should be formatted, and the results it returns.
eclscheduler
Monitors workunits that are waiting for events, and updates them when those events occur.
eclagent/Roxie
Process the different workflow actions, and workflow code.
hThor/Roxie/Thor
Execute graphs within the workflow items.
Dali
This database is used to store the state of the workunit state.
The following ECL will be used as an example for the rest of the discussion. It is a very simple search that takes a string parameter 'searchName', which is the name of the person to search for, and returns the matching records from an index. It also outputs the word 'Done!' as a separate result.
This section outlines the different sections in a workunit. This is followed by a walk-through of the stages that take place when a workunit is executed, together with a more detailed explanation of the workunit contents.
The workflow is the highest level of control within a workunit. It is used for two related purposes:
Scheduling The HPCC system allows ECL code to be executed when certain events occur - e.g. every hour or when files are uploaded to a landing zone. Each piece of ECL code that is triggered by an external event creates a separate workflow action. This allows each of those events to be processed independently.
Splitting up queries. There are situations where it is useful to break parts of an ECL query into independent sections. The simplest example is the PERSIST workflow operation, which allows results to be shared between different work units. Each workflow operation creates one (or sometimes more) independent workflow items, which are then connected together.
Each piece of independent ECL is given a unique workflow id (wfid). Often workflow items need to be executed in a particular order, e.g. ensuring a persist exists before using it, which is managed with dependencies between different workflow items.
Our example above generates the following XML entry in the workunit:
This contains two workflow items. The first workflow item (wfid=1) ensures that the stored value has a default value if it has not been supplied. The second item (with wfid=2) is the main code for the query. This has a dependency on the first workflow item because the stored variable needs to be intialised before it is executed.
The generated code contains a class instance that is used for executing the code associated with the workflow items. It is generated at the end of the main C++ module. E.g.:
cpp
struct MyEclProcess : public EclProcess {
+ virtual int perform(IGlobalCodeContext * gctx, unsigned wfid) {
+ ....
+ switch (wfid) {
+ case 1U:
+ ... code for workflow item 1 ...
+ case 2U:
+ ... code for workflow item 2 ...
+ break;
+ }
+ return 2U;
+ }
+};
The main element is a switch statement inside the perform() function that allows the workflow engines to execute the code associated with a particular workflow item.
There is also an associated factory function that is exported from the dll, and is used by the engines to create instances of the class:
Most of the work executing a query involves processing dataset operations, which are implemented as a graph of activities. Each graph is represented in the workunit as an xml graph structure (currently it uses the xgmml format). The graph xml includes details of which types of activities are required to be executed, how they are linked together, and any other dependencies.
This graph contains a single subgraph (node id=2) that contains two activities - an index read activity and an output result activity. These activities are linked by a single edge (id "2_0"). The details of the contents are covered in the section on executing graphs below.
Each activity has a corresponding class instance in the generated code, and a factory function for creating instances of that class:
cpp
struct cAc2 : public CThorIndexReadArg {
+ ... Implementation of the helper for activity #2 ...
+};
+extern "C" ECL_API IHThorArg * fAc2() { return new cAc2; }
+
+struct cAc3 : public CThorWorkUnitWriteArg {
+ ... Implementation of the helper for activity #3 ...
+};
+extern "C" ECL_API IHThorArg * fAc3() { return new cAc3; }
The helper class for an activity implements the interface that is required for that particular kind. (The interfaces are defined in rtl/include/eclhelper.hpp - further details below.)
The are several other items, detailed below, that are logically associated with a workunit. The information may be stored in the workunit dll or in various other location e.g. Dali, Sasha or Cassandra. It is all accessed through the IWorkUnit interface in common/workunit/workunit.hpp that hides the implementation details. For instance information generated at runtime cannot by definition be included in the workunit dll.
Many queries contain input parameters that modify their behaviour. These correspond to STORED definitions in the ECL. Our example contains a single string "searchName", so the workunit contains a single input parameter:
in our example there are two - the dataset of results and the text string "Done!". The values of the results for a query are associated with the workunit. (They are currently saved in dali, but this may change in version 6.0.)
Other statistics and timings created when running the query are stored in the runtime copy of the workunit. (Statistics for graph elements are stored in a different format from global statistics, but the IWorkUnit interface ensures the implementation details are hidden.)
It is possible to include other user-defined resources in the workunit dll - e.g. web pages, or dashboard layouts. I have to confess I do not understand them... ??Tony please provide some more information....!
Once a workunit has been compiled to a dll it is ready to be executed. Execution can be triggered in different ways, E.g.:
The ECL for a query is submitted to esp
A workunit entry, containing the ECL, is created in dali and added to an eclccserver queue.
An eclccserver instance removes the workunit form the queue, and compiles the ECL to a workunit dll.
The dali workunit entry is updated with the information from the workunit dll.
The dali workunit is added to the agent execution queue associated with the target cluster.
The associated engine (actually agentexec for hThor and Thor) pulls a query form the queue and executes it.
A query is submitted and published with a name. Another request is then submitted to execute this previously compiled query.
A workunit entry, containing the ECL, is created in dali and added to an eclccserver queue.
An eclccserver instance removes the workunit form the queue, and compiles the ECL to a workunit dll.
There is a 'query set' for each combination of query name and the target cluster. The new workunit dll is added to the appropriate query set, and marked as the current active implementation.
Later, a query that references a named query is submitted to esp.
The name and target cluster are mapped via the query set to the active implementation, and a workunit instance is created from the active workunit dll.
The workunit is added to a roxie or eclagentexec queue ready to be executed.
The associated engine pulls a query form the queue and executes it.
A query is compiled as a stand alone executable. The executable is then run.
eclcc is executed on the command line without the -shared command line option.
The resulting executable is run. The engine used to execute the query depends on the -platform parameter supplied to eclcc.
Most queries create persistent workunits in dali and then update those workunits with results as they are calculated, however for some roxie queries (e.g. in a production system) the execution workunits are transient.
The following walk-through details the main stages executing a query, and the effect each of the query elements has.
The system uses several inter-process queues to communicate between the different components in the system. These queues are implemented by dali. Components can subscribe to one or more queues, and receive notifications when entries are avaialable.
Some example queues are:
<cluster>.eclserver - workunits to be compiled
<cluster>.roxie - workunits to execute on roxie
<cluster>.thor - graphs to execute on thor
<cluster>.eclscheduler - workunits that need to wait for events
<cluster>.agent - workunits to be executed on hthor or thor.
dfuserver_queue - dfu workunits for sprays/file copies etc.
When a workunit is ready to be run, the workflow controls the flow of execution. The workflow engine (defined in common/workunit/workflow.cpp) is responsible for determining which workflow item should be executed next.
The workflow for Thor and hThor jobs is coordinated by eclagent, while roxie includes the workflow engine in its process. The eclscheduler also uses the workflow engine to process events and mark workflow items ready for execution.
eclagent, or roxie calls the createProcess() function from the workunit dll to create an instance of the generated workflow helper class, and passes it to the workflow engine. The workflow engine walks the workflow items to find any items that are ready to be executed (have the state "reqd" - i.e. required). If a required workflow item has dependencies on other child workflow items then those children are executed first. Once all dependencies have executed successfully the parent workflow item is executed. The example has the following workflow entries:
Item 2 has a state of "reqd", so it should be evaluated now. Item 2 has a dependency on item 1, so that must be evaluated first. This is achieved by calling MyEclProcess::perform() on the object that was previously created from the workunit dll, passing in wfid = 1. That will execute the following code:
cpp
switch (wfid) {
+ case 1U:
+ if (!gctx->isResult("searchname",4294967295U)) {
+ ctx->setResultString("searchname",4294967295U,5U,"Smith");
+ }
+ break;
+ break;
+}
This checks if a value has been provided for the input parameter, and if not assigns a default value of "Smith". The function returns control to the workflow engine. With the dependencies for wfid 2 now satisfied, the generated code for that workflow id is now executed:
Most of the work for this workflow item involves executing graph1 (by calling back into eclagent/roxie). However, the code also directly sets another result. This is fairly typical - the code inside MyProcess::perform often combines evaluating scalar results, executing graphs, and calling functions that cannot (currently) be called inside a graph (e.g. those involving superfile transactions).
Once all of the required workflow items are executed, the workunit is marked as completed. Alternatively, if there are workflow items that are waiting to be triggered by an event, the workunit will be passed to the scheduler, which will keep monitoring for events.
Note that most items in the xml workflow begin in the state WFStateNull. This means that it is valid to execute them, but they haven't been executed yet. Typically, only a few items begin with the state WFStateReqd.
There are various specialised types of workflow items - e.g. sequential, wait, independent, but they all follow the same basic approach of executing dependencies and then executing that particular item.
Most of the interesting work in an ECL query is done within a graph. The call ctx->executeGraph will either execute the graph locally (in the case of hthor and roxie), or add the workunit onto a queue (for Thor). Whichever happens, control will pass to that engine.
Each item mode/type can affect the dependency structure of the workflow:
Sequential/Ordered
The workflow structure for sequential and ordered is the same. An item is made to contain all of the actions in the statement. This is achieved by making each action a dependency of this item. The dependencies, and consequently the actions, must be executed in order. An extra item is always inserted before each dependency. This means that if other statements reference the same dependency, it will only be performed once.
Persist
When the persist workflow service is used, two items are created. One item contains the graphs that perform the expression defined in ECL. It also stores the wfid for the second item. The second item is used to determine whether the persist is up to date.
Condition (IF)
The IF function has either 2 or 3 arguments: the expression, the trueresult, and sometimes the falseresult. For each argument, a workflow item is created. These items are stored as dependencies to the condition, in the order stated above.
Contingency (SUCCESS/FAILURE)
When a contingency clause is defined for an attribute, the attribute item stores the wfid of the contingency. If both success and failure are used, then the item will store the wfid of each contingency. The contingency is composed of items, just like the larger query.
Recovery
When a workflow item fails, if it has a recovery clause, the item will be re-executed. The clause contains actions defined by the programmer to remedy the problem. This clause is stored differently to SUCCESS/FAILURE, in that the recovery clause is a dependency of the failed item. In order to stop the recovery clause from being executed like the other dependencies, it is marked with WFStateSkip.
Independent
This specifier is used when a programmer wants to common up code for the query. It prevents the same piece of code from being executed twice in different contexts. To achieve this, an extra item is inserted between the expression and whichever items depend on it. This means that although the attribute can be referenced several times, it will only be executed once.
All the engines (roxie, hThor, Thor) execute graphs in a very similar way. The main differences are that hThor and Thor execute a sub graph at a time, while roxie executes a complete graph as one. Roxie is also optimized to minimize the overhead of executing a query - since the same query tends to be run multiple times. This means that roxie creates a graph of factory objects and those are then used to create the activities. The core details are the same for each of them though.
Each graph (e.g. graph1) consists of 1 or more subgraphs (in the example above, node id=1). Each of those subgraphs contains 1 or more activities (node id=2, node id=3).
The xml for each activity might contain the following information:
A unique id (e.g. id="2").
The "kind" of the activity, e.g. <att name="_kind" value="77"/>. The value is an item from the enum ThorActivityKind in eclhelper.hpp.
The ECL that created the activity. E.g. <att name="ecl" value="...">
The identifier of the ecl definition. E.g. <att name="name" value="results"/>
Location (e.g. file, line number, column) of the original ECL. E.g. <att name="definition" value="workuniteg1.ecl(3,1)"/>
Meta information the code generator has deduced about the activity output. Examples include the record size, expected number of rows, sort order etc. E.g. <att name="recordSize" value="120"/>
Hints, which are used for fine control of options for a particular activity (e.g,, the number of threads to use while sorting).
Record counts and stats once the job has executed. (These are logically associated with the activities in the graph, but stored separately.)
Graphs also contain edges that can take one of 3 forms:
Edges within graphs
: These are used to indicate how the activities are connected. The source activity is used as the input to the target activity. These edges have the following format:
<edge id="<source-activity-id>_<output-count>" source="<source-activity-id>" target="<target-activity-id">
+
+There is only one edge in our example workunit: \\<edge id="2\\_0"
+source="2" target="3"/\\>.
+
Edges between graphs
: These are used to indicate direct dependencies between activities. For instance there will be an edge connecting the activity that writes a spill file to the activity that reads it. These edges have the following format:
<edge id="<source-activity-id>_<target-activity-id>" source="<source-subgraph-id>" target="<target-subgraph-id>"
+ <att name="_sourceActivity" value="<source-activity-id>"/>
+ <att name="_targetActivity" value="<target-activity-id>"/>
+ </edge>
+
+Roxie often optimizes spilled datasets and treats these edges as
+equivalent to the edges between activities.
+
Other dependencies.
: These are similar to the edges between graphs, but they are used for values that are used within an activity. For instance one part of the graph may calculate the maximum value of a column, and another activity may filter out all records that do not match that maximum value. The format is the same as the edges between graphs except that the edge contains the following attribute:
<att name="_dependsOn" value="1"/>
+
Each activity in a graph also has a corresponding helper class instance in the generated code. (The name of the class is "cAc" followed by the activity number, and the exported factory method is "fAc" followed by the activity number.) Each helper class implements a specialised interface (derived from IHThorArg) - the particular interface is determined by the value of the "_kind" attribute for the activity.
The contents of file rtl/include/eclhelper.hpp is key to understanding how the generated code relates to the activities. Each kind of activity requires a helper class that implements a specific interface. The helpers allow the engine to tailor the generalised activity implementation to the the particular instance - e.g. for a filter activity whether a row should be included or excluded. The appendix at the end of this document contains some further information about this file.
The classes in the generated workunits are normally derived from base implementations of those interfaces (which are implemented in rtl/include/eclhelper_base.hpp). This reduces the size of the generated code by providing default implementations for various functions.
For instance the helper for the index read (activity 2) is defined as:
a) onCreate() - common to all activities. It is called by the engine when the helper is first created, and allows the helper to cache information that does not change - in this case the name that is being searched for. b) getFileName() - determines the name of the index being read. c) createSegmentMonitors() - defines which columns to filter, and which values to match against. d) transform() - create the record to return from the activity. It controls which columns should be included from the index row in the output. (In this case all.)
To execute a graph, the engine walks the activities in the graph xml and creates, in memory, a graph of implementation activities.
For each activity, the name of the helper factory is calculated from the activity number (e.g. fAc2 for activity 2). That function is imported from the loaded dll, and then called to create an instance of the generated helper class - in this case cAc2.
The engine then creates an instance of the class for implementing the activity, and passes the previously created helper object to the constructor. The engine uses the _kind attribute in the graph to determine which activity class should be used. E.g. In the example above activity 2 has a _kind of 77, which corresponds to TAKindexread. For an index-read activity roxie will create an instance of CRoxieServerIndexReadActivity. (The generated helper that is passed to the activity instance will implement IHThorIndexReadArg). The activity implementations may also extract other information from the xml for the activity - e.g. hints. Once all the activities are created the edge information is used to link inputs activities to output activities and add other dependencies.
Note: Any subgraph that is marked with <att name="rootGraph" value="1"/> is a root subgraph. An activity within a subgraph that has no outputs is called a 'sink' (and an activity without any inputs is called a 'source').
Executing a graph involves executing all the root subgraphs that it contains. All dependencies of the activities within the subgraph must be executed before a subgraph is executed. To execute a subgraph, the engine executes each of the sink activities on separate threads, and then waits for each of those threads to complete. Each sink activity lazily pulls input rows on demand from activities further up the graph, processes them and returns when complete.
(If you examine the code you will find that this is a simplification. The implementation for processing dependencies is more fine grained to ensure IF datasets, OUPUT(,UPDATE) and other ECL constructs are executed correctly.)
In our example the execution flows as follows:
Only a single root subgraph, so need to execute that.
The engine will execute activity 3 - the workunit-write activity (TAKworkunitwrite).
The workunit-write activity will start its input.
The index-read activity will call the helper functions to obtain the filename, resolve the index, and create the filter.
The workunit-write activity requests a row from its input.
The index-read finds the first row, and calls the helper's transform() method to create an output row.
The workunit-write activity persists the row to a buffer (using the serializer provided by the IOutputMetaData interface implemented by the class mx1).
Back to step 5, workunit-write reading a row from its input, until end of file is returned (notified as two consecutive NULL rows.
Workunit-write commits the results and finishes.
The execution generally switches back and forth between the code in the engines, and the members of the generated helper classes.
There are some other details of query execution that are worth highlighting:
Child Queries
: Some activities perform complicated operations on child datasets of the input rows. E.g. remove all duplicate people who are marked as living at this address. This will create a "child query" in the graph - i.e. a nested graph within a subgraph, which may be executed each time a new input row is processed by the containing activity. (The graph of activities for each child query is created at the same time as the parent activity. The activity instances are reinitialised and re-executed for each input row processed by the parent activity to minimise the create-time overhead.)
Other helper functions
: The generated code contains other functions that are used to describe the meta information for the rows processed within the graph. E.g. the following class describes the output from the index read activity:
This represents a fixed size row that occupies 120 bytes. The object
+returned by the queryTypeInfo() function provides information about
+the types of the fields:
+
I.e. a string column, length 40 called "name", followed by a
+string column, length 80 called "address". The interface
+IOutputMetaData in eclhelper.hpp is key to understanding how the
+rows are processed.
+
Inline dataset operations
: The rule mentioned at the start - that the generated code does not contain any knowledge of how to perform a particular dataset operation - does have one notable exception. Some operations on child datasets are very simple to implement, and more efficient if they are implemented using inline C++ code. (The generated code is smaller, and they avoid the overhead of setting up a child graph.) Examples include filtering and aggregating column values from a child dataset.
The full code in the different engines is more complicated than the simplified process outlined above, especially when it comes to executing dependencies, but the broad outline is the same.
More information on the work done in the code generator to create the workunit can be found in ecl/eclcc/DOCUMENTATION.rst.
The C++ code can be generated as a single C++ file or multiple files. The system defaults to multiple C++ files, so that they can be compiled in parallel (and to avoid problems some compilers have with very large files). When multipe C++ files are generated the metadata classes and workflow classes are generated in the main module, and the activities are generated in the files suffixed with a number. It may be easier to understand the generated code if it is in one place. In which case, compile your query with the option -fspanMultipleCpp=0. Use -fsaveCppTempFiles to ensure the C++ files are not deleted (the C++ files will appear as helpers in the workunit details).
: The interface that is used by the workflow engine to execute the different workflow items in the generated code.
ThorActivityKind
: This enumeration contains one entry for each activity supported by the engines.
ICodeContext
: This interface is implemented by the engine, and provides a mechanism for the generated code to call back into the engine. For example resolveChildQuery() is used to obtain a reference to a child query that can then be executed later.
IOutputMetaData
: This interface is used to describe any meta data associated with the data being processed by the queries.
IHThorArg
: The base interface for defining information about an activity. Each activity has an associated interface that is derived from this interface. E.g. each instance of the sort activity will have a helper class implementing IHThorSortArg in the generated query. There is normally a corresponding base class for each interface in eclhelper_base.hpp that is used by the generated code e.g. CThorSortArg.
ARowBuilder
: This abstract base class is used by the transform functions to reserve memory for the rows that are created.
IEngineRowAllocator
: Used by the generated code to allocate rows and rowsets. Can also be used to release rows (or call the global function rtlReleaseRow()).
IGlobalCodeContext
: Provides access to functions that cannot be called inside a graph - i.e. can only be called from the global workflow code. Most functions are related to the internal implementation of particular workflow item types (e.g. persists).
: An activity is the basic unit of dataset processing implemented by the engines. Each activity corresponds to a node in the thor execution graph. Instances of the activities are connnected together to create the graph.
dll
: A dynamically loaded library. These correspond to shared objects in Linux (extension '.so'), dynamic libraries in Max OS X ('.dylib'), and dynamic link libraries in windows ('.dll').
superfile
: A composite file which allows a collection of files to be treated as a single compound file.
The XML for a workunit can be viewed on the XML tag in eclwatch, or generated by compiling the ECL using the -wu option with eclcc. Alternatively eclcc -b -S can be used to generate the XML and the C++ at the same time (the output filenames are derived from the input name).
`,170)]))}const o=i(h,[["render",e]]);export{g as __pageData,o as default};
diff --git a/assets/devdoc_docs_ContributeDocs.md.B6UEh3AF.js b/assets/devdoc_docs_ContributeDocs.md.B6UEh3AF.js
new file mode 100644
index 00000000000..2cce31b1926
--- /dev/null
+++ b/assets/devdoc_docs_ContributeDocs.md.B6UEh3AF.js
@@ -0,0 +1 @@
+import{_ as t,c as o,a3 as n,o as a}from"./chunks/framework.DkhCEVKm.js";const p=JSON.parse('{"title":"Contributing Documentation to the HPCC Systems Platform Project","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/docs/ContributeDocs.md","filePath":"devdoc/docs/ContributeDocs.md","lastUpdated":1731340314000}'),i={name:"devdoc/docs/ContributeDocs.md"};function r(s,e,l,d,u,h){return a(),o("div",null,e[0]||(e[0]=[n('
Contributing Documentation to the HPCC Systems Platform Project
This document is intended for anyone that wants to contribute documentation to our project. The first audience is platform developers, so we can streamline the process of documenting new features. However, these guidelines apply to anyone who wants to contribute to any of our documentation (Language Reference, Programmer’s Guide, etc.).
This set of guidelines should help you understand the information needed to create sufficient documentation for any new feature you add. The good news is that we are all here to help you and support your writing efforts. We will help by advising you along the way, and by reviewing and editing your submissions.
Documenting a New Software Feature--Required and Optional Components
When you create a new feature or function, clear documentation is crucial for both internal teams and external users. You worked hard on the feature, so it deserves proper notice and usage.
Contributions to the platform are always welcome, and we strongly encourage developers and users to contribute documentation.
You can contribute on many levels:
Developer Notes
End user “Readmes” in the form of MD files in the GitHub repository
Blogs
Formal documentation
Regardless of the form you are planning to deliver, here are the required and optional components to include in a document.
Tip: VS Code is very good at editing MD files. There is a built-in Preview panel available to be able to see the rendered form.
In addition, GitHub Copilot is MD-aware and can help you write and format. For example, you can ask the Copilot, “How can I align the content within the cells of my Markdown table?” GitHub copilot will show you the alignment options.
What it is: Briefly describe the feature's purpose and the problem it solves.
Why it matters: Explain the value proposition for users and the overall impact on the software.
Target audience: Specify who this feature is designed for (for example, all users or specific user roles).
Use Cases: Provide concrete examples of how a user might leverage this feature in a real-world scenario.
Installation and Configuration: Details on how to install and basic setup for use, if needed. This must include any system requirements or dependencies.
User Guide / Functionality
How it works: Provide a task-oriented, step-by-step guide for using the feature. If possible, include screenshots for visual learners.
Tips, Tricks, and Techniques: Explain any shortcuts or clever uses of the feature that may be non-obvious.
Inputs and Outputs: Detail the information users need to provide to the feature and the format of the results.
Error Handling: Explain what happens if users encounter errors and how to troubleshoot common issues.
Limitations and Considerations:
Limitations: Acknowledge any restrictions or boundaries associated with the feature's functionality.
Detailed configuration options: If the feature offers advanced settings or customizations, provide in-depth instructions for experienced users. This is similar to the way some options command line program's usage are only displayed using the verbose option.
API Reference (for technical audiences)
Technical specifications: For features with an API component, include detailed API reference documentation for developers integrating it into their applications.
FAQs
Frequently Asked Questions: Address any commonly anticipated user questions about the feature to pre-empt confusion.
Additional Resources
Links to related documentation: Include links to relevant documentation for features that interact with this new addition.
Tutorials: Consider creating tutorials for a more interactive learning experience.
Videos: Consider requesting that a video be made to provide a more interactive visual learning experience. You should provide a simple script or outline of what should be shown in the video.
Target your audience: Tailor the level of detail and technical jargon you use based on whether the documentation is for developers or end-users.
Clarity and Conciseness: Use clear, concise language and maintain a consistent structure for easy navigation. Always use present tense, active voice. Remember, you’re writing for users and programmers, not academics, so keep it simple and straightforward. See the HPCC Style Guide for additional guidance.
Visual Aids: Screenshots, diagrams, and flowcharts can significantly enhance understanding. A picture can communicate instantly what a thousand words cannot.
Maintain and Update: Regularly review and update documentation as the feature evolves or based on user feedback.
By following these guidelines and including the required and optional components, you can create comprehensive documentation that empowers users and streamlines the adoption of your new software feature.
The boundary between a developer's responsibilities and the documentation team’s responsibility is not cast in stone. However, there are some guidelines that can help you decide what your responsibility is. Here are some examples:
Changing the default value of a configuration setting
This typically needs a simple one or two word change in the area of the documentation where that setting is documented. However, the change could impact existing deployments or existing code and therefore it might also require a short write-up for the Red Book and/or Release Announcement. If the setting is used by both bare-metal and containerized, you should provide information about how the new setting is used in each of those deployments.
Adding or modifying a Language keyword, Standard Library function, or command line tool action
This needs some changes to existing documentation so the best way to provide the information is in a documentation Jira issue. If it s a new keyword, function, or action, a brief overview should be included. For a Standard Library function, the developer should update the Javadoc comment in the appropriate ECL file. For a command line tool change, the developer should update the Usage section of the code.
This is a candidate for either an MD file, a blog, or both. Since there should have been some sort of design specification document, that could easily be repurposed as a good start for this.
A feature/function that is only used internally to the system
Since this is information that is probably only of interest to other developers, a write-up in the form of an MD file in the repo is the best approach. If it affects end-users or operations, then a more formal document or a blog might be a good idea. If it affects existing deployments or existing code, then a Red Book notice might also be needed.
New tests are frequently added and the regression suite readme files should be updated at the same time. If the tests are noteworthy, we could add a mention in the Platform Release Notes.
In general, it makes sense to keep simple documentation near the code. For example, a document about ECL Agent should go in the ECLAgent folder. However, there are times where that is either not possible or a document may cover more than one component. In those cases, there are a few options as shown below.
devdoc: This is a general folder for any developer document.
devdoc/docs: This is a folder for documents about documentation.
devdoc/userdoc: This is a collection of docs targeted toward the end-user rather than developers.
This is primarily for informal documents aimed at end-users. This info can and should be incorporated into the formal docs. devdoc/userdoc/troubleshoot: Information related to troubleshooting particular components
devdoc/userdoc/azure: Useful information about Azure Cloud portal and cli
devdoc/userdoc/roxie: Useful information for running roxie
devdoc/userdoc/thor: Useful information for running thor
devdoc/userdoc/blogs: COMING SOON: Location and storage of original text for blogs. It also has docs with guidelines and instructions on writing Blogs
You can include your documentation with your code in a Pull Request or create a separate Jira and Pull Request for the documentation. This depends on the size of the code and doc. For a large project or change, a separate Pull request for the documentation is better. This might allow the code change to be merged faster.
For minor code changes, for example the addition of a parameter to an existing ECL keyword, you can request a documentation change in a Jira issue. You should provide sufficient details in the Jira.
For example, If you add an optional parameter named Foo, you should provide details about what values can be passed in through the Foo parameter and what those values mean. You should also provide the default value used if the parameter is omitted.
',45)]))}const g=t(i,[["render",r]]);export{p as __pageData,g as default};
diff --git a/assets/devdoc_docs_ContributeDocs.md.B6UEh3AF.lean.js b/assets/devdoc_docs_ContributeDocs.md.B6UEh3AF.lean.js
new file mode 100644
index 00000000000..2cce31b1926
--- /dev/null
+++ b/assets/devdoc_docs_ContributeDocs.md.B6UEh3AF.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as o,a3 as n,o as a}from"./chunks/framework.DkhCEVKm.js";const p=JSON.parse('{"title":"Contributing Documentation to the HPCC Systems Platform Project","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/docs/ContributeDocs.md","filePath":"devdoc/docs/ContributeDocs.md","lastUpdated":1731340314000}'),i={name:"devdoc/docs/ContributeDocs.md"};function r(s,e,l,d,u,h){return a(),o("div",null,e[0]||(e[0]=[n('
Contributing Documentation to the HPCC Systems Platform Project
This document is intended for anyone that wants to contribute documentation to our project. The first audience is platform developers, so we can streamline the process of documenting new features. However, these guidelines apply to anyone who wants to contribute to any of our documentation (Language Reference, Programmer’s Guide, etc.).
This set of guidelines should help you understand the information needed to create sufficient documentation for any new feature you add. The good news is that we are all here to help you and support your writing efforts. We will help by advising you along the way, and by reviewing and editing your submissions.
Documenting a New Software Feature--Required and Optional Components
When you create a new feature or function, clear documentation is crucial for both internal teams and external users. You worked hard on the feature, so it deserves proper notice and usage.
Contributions to the platform are always welcome, and we strongly encourage developers and users to contribute documentation.
You can contribute on many levels:
Developer Notes
End user “Readmes” in the form of MD files in the GitHub repository
Blogs
Formal documentation
Regardless of the form you are planning to deliver, here are the required and optional components to include in a document.
Tip: VS Code is very good at editing MD files. There is a built-in Preview panel available to be able to see the rendered form.
In addition, GitHub Copilot is MD-aware and can help you write and format. For example, you can ask the Copilot, “How can I align the content within the cells of my Markdown table?” GitHub copilot will show you the alignment options.
What it is: Briefly describe the feature's purpose and the problem it solves.
Why it matters: Explain the value proposition for users and the overall impact on the software.
Target audience: Specify who this feature is designed for (for example, all users or specific user roles).
Use Cases: Provide concrete examples of how a user might leverage this feature in a real-world scenario.
Installation and Configuration: Details on how to install and basic setup for use, if needed. This must include any system requirements or dependencies.
User Guide / Functionality
How it works: Provide a task-oriented, step-by-step guide for using the feature. If possible, include screenshots for visual learners.
Tips, Tricks, and Techniques: Explain any shortcuts or clever uses of the feature that may be non-obvious.
Inputs and Outputs: Detail the information users need to provide to the feature and the format of the results.
Error Handling: Explain what happens if users encounter errors and how to troubleshoot common issues.
Limitations and Considerations:
Limitations: Acknowledge any restrictions or boundaries associated with the feature's functionality.
Detailed configuration options: If the feature offers advanced settings or customizations, provide in-depth instructions for experienced users. This is similar to the way some options command line program's usage are only displayed using the verbose option.
API Reference (for technical audiences)
Technical specifications: For features with an API component, include detailed API reference documentation for developers integrating it into their applications.
FAQs
Frequently Asked Questions: Address any commonly anticipated user questions about the feature to pre-empt confusion.
Additional Resources
Links to related documentation: Include links to relevant documentation for features that interact with this new addition.
Tutorials: Consider creating tutorials for a more interactive learning experience.
Videos: Consider requesting that a video be made to provide a more interactive visual learning experience. You should provide a simple script or outline of what should be shown in the video.
Target your audience: Tailor the level of detail and technical jargon you use based on whether the documentation is for developers or end-users.
Clarity and Conciseness: Use clear, concise language and maintain a consistent structure for easy navigation. Always use present tense, active voice. Remember, you’re writing for users and programmers, not academics, so keep it simple and straightforward. See the HPCC Style Guide for additional guidance.
Visual Aids: Screenshots, diagrams, and flowcharts can significantly enhance understanding. A picture can communicate instantly what a thousand words cannot.
Maintain and Update: Regularly review and update documentation as the feature evolves or based on user feedback.
By following these guidelines and including the required and optional components, you can create comprehensive documentation that empowers users and streamlines the adoption of your new software feature.
The boundary between a developer's responsibilities and the documentation team’s responsibility is not cast in stone. However, there are some guidelines that can help you decide what your responsibility is. Here are some examples:
Changing the default value of a configuration setting
This typically needs a simple one or two word change in the area of the documentation where that setting is documented. However, the change could impact existing deployments or existing code and therefore it might also require a short write-up for the Red Book and/or Release Announcement. If the setting is used by both bare-metal and containerized, you should provide information about how the new setting is used in each of those deployments.
Adding or modifying a Language keyword, Standard Library function, or command line tool action
This needs some changes to existing documentation so the best way to provide the information is in a documentation Jira issue. If it s a new keyword, function, or action, a brief overview should be included. For a Standard Library function, the developer should update the Javadoc comment in the appropriate ECL file. For a command line tool change, the developer should update the Usage section of the code.
This is a candidate for either an MD file, a blog, or both. Since there should have been some sort of design specification document, that could easily be repurposed as a good start for this.
A feature/function that is only used internally to the system
Since this is information that is probably only of interest to other developers, a write-up in the form of an MD file in the repo is the best approach. If it affects end-users or operations, then a more formal document or a blog might be a good idea. If it affects existing deployments or existing code, then a Red Book notice might also be needed.
New tests are frequently added and the regression suite readme files should be updated at the same time. If the tests are noteworthy, we could add a mention in the Platform Release Notes.
In general, it makes sense to keep simple documentation near the code. For example, a document about ECL Agent should go in the ECLAgent folder. However, there are times where that is either not possible or a document may cover more than one component. In those cases, there are a few options as shown below.
devdoc: This is a general folder for any developer document.
devdoc/docs: This is a folder for documents about documentation.
devdoc/userdoc: This is a collection of docs targeted toward the end-user rather than developers.
This is primarily for informal documents aimed at end-users. This info can and should be incorporated into the formal docs. devdoc/userdoc/troubleshoot: Information related to troubleshooting particular components
devdoc/userdoc/azure: Useful information about Azure Cloud portal and cli
devdoc/userdoc/roxie: Useful information for running roxie
devdoc/userdoc/thor: Useful information for running thor
devdoc/userdoc/blogs: COMING SOON: Location and storage of original text for blogs. It also has docs with guidelines and instructions on writing Blogs
You can include your documentation with your code in a Pull Request or create a separate Jira and Pull Request for the documentation. This depends on the size of the code and doc. For a large project or change, a separate Pull request for the documentation is better. This might allow the code change to be merged faster.
For minor code changes, for example the addition of a parameter to an existing ECL keyword, you can request a documentation change in a Jira issue. You should provide sufficient details in the Jira.
For example, If you add an optional parameter named Foo, you should provide details about what values can be passed in through the Foo parameter and what those values mean. You should also provide the default value used if the parameter is omitted.
',45)]))}const g=t(i,[["render",r]]);export{p as __pageData,g as default};
diff --git a/assets/devdoc_docs_DocTemplate.md.DO145SYO.js b/assets/devdoc_docs_DocTemplate.md.DO145SYO.js
new file mode 100644
index 00000000000..8ec8046024c
--- /dev/null
+++ b/assets/devdoc_docs_DocTemplate.md.DO145SYO.js
@@ -0,0 +1 @@
+import{_ as t,c as a,a3 as o,o as r}from"./chunks/framework.DkhCEVKm.js";const h=JSON.parse('{"title":"Feature Documentation Template","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/docs/DocTemplate.md","filePath":"devdoc/docs/DocTemplate.md","lastUpdated":1731340314000}'),i={name:"devdoc/docs/DocTemplate.md"};function n(s,e,l,u,c,p){return r(),a("div",null,e[0]||(e[0]=[o('
Use this template to create new documentation for a feature, function, or process. Not every feature will require all six sections. Delete the ones that are not applicaple.
[Address common issues or errors that users may encounter while using the new feature, along with possible solutions or workarounds.]
',14)]))}const m=t(i,[["render",n]]);export{h as __pageData,m as default};
diff --git a/assets/devdoc_docs_DocTemplate.md.DO145SYO.lean.js b/assets/devdoc_docs_DocTemplate.md.DO145SYO.lean.js
new file mode 100644
index 00000000000..8ec8046024c
--- /dev/null
+++ b/assets/devdoc_docs_DocTemplate.md.DO145SYO.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as a,a3 as o,o as r}from"./chunks/framework.DkhCEVKm.js";const h=JSON.parse('{"title":"Feature Documentation Template","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/docs/DocTemplate.md","filePath":"devdoc/docs/DocTemplate.md","lastUpdated":1731340314000}'),i={name:"devdoc/docs/DocTemplate.md"};function n(s,e,l,u,c,p){return r(),a("div",null,e[0]||(e[0]=[o('
Use this template to create new documentation for a feature, function, or process. Not every feature will require all six sections. Delete the ones that are not applicaple.
[Address common issues or errors that users may encounter while using the new feature, along with possible solutions or workarounds.]
',14)]))}const m=t(i,[["render",n]]);export{h as __pageData,m as default};
diff --git a/assets/devdoc_docs_HPCCStyleGuide.md.ttTlrCG0.js b/assets/devdoc_docs_HPCCStyleGuide.md.ttTlrCG0.js
new file mode 100644
index 00000000000..bb353876463
--- /dev/null
+++ b/assets/devdoc_docs_HPCCStyleGuide.md.ttTlrCG0.js
@@ -0,0 +1 @@
+import{_ as t,c as i,a3 as o,o as s}from"./chunks/framework.DkhCEVKm.js";const d=JSON.parse('{"title":"HPCC Writing Style Guide","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/docs/HPCCStyleGuide.md","filePath":"devdoc/docs/HPCCStyleGuide.md","lastUpdated":1731340314000}'),a={name:"devdoc/docs/HPCCStyleGuide.md"};function n(r,e,l,p,u,h){return s(),i("div",null,e[0]||(e[0]=[o('
We strive to maintain a consistent voice. These guidelines can help your writing match that voice.
Use present tense, active voice. Documentation should be you speaking directly to the reader. Simply tell them what to do.
This example sentence: "The user selects the file menu." is passive voice. You wouldn’t say it that way in a conversation.
You should use active voice wording, such as: "Select the file menu".
Similarly, instructions like these are active voice: "Press the button" or "Submit the file".
Documentation is you instructing the user. Just tell them what to do.
Be Brief and keep it simple. Be efficient with your words. Keep sentences short and concise. Keep paragraphs short as well. Use just a few sentences per paragraph. Use simple words wherever possible and try to avoid lengthy explanations.
Consistency: Be consistent. Use the same voice across all documents. Use the same term, use the same spelling, punctuation, etc. Follow the conventions in this guide.
When writing formal documentation that is more than a new feature announcement, do not refer to a new feature, or coming features as such, this does not hold up well over time.
Do not refer to how things were done in the past. For example, "in the past we had to do X-Y-Z steps and now we no longer have to. This ‘new feature’ can do it in one step, Z". This only adds potential confusion. Just instruct on exactly what needs to be done now, using words efficiently as possible, so for this example just say “perform step Z”
There are many terms specific to HPCC Systems® and the HPCC Systems platform. Use the following style guide for word usage and capitalization guidelines to use when referring to system components or other HPCC-specific entities. Maintain consistent usage throughout all docs.
Officially and legally the organizaion's name is HPCC Systems® and it is a registered trademark.
You should always refer to the platform as the HPCC Systems® platform and the registration mark ® should appear in the first and most prominent mention of the name.
While it is acceptable to use the ® anywhere in a document, it is required to be used in the first and most prominent mention - so the average reader will be aware. Any usage after that first and most prominent is optional.
Components and Tools
HPCC Systems Platform
Dali
Sasha
Thor
hThor
ROXIE
DFU Server (Distributed File Utility Server)
ESP Server (Enterprise Server Platform)
ESP Services
WsECL
ECL Watch
ECL Server
ECLCC Server
ECL Agent
ECL IDE
ECL Plug-in for Eclipse
ECL Playground
LDAP
dafilesrv
VS Code (no hyphen)
ECL Language Extension for VS Code
Configuration Manager (not ConfigMgr or ConfigManager)
Note: when referring to the startup command, use configmgr (always lowercase)
To “assure” a person of something is to make him or her confident of it.
According to the Associated Press style, to “ensure” that something happens is to make certain that it does, and to “insure” is to issue an insurance policy.
Other authorities, however, consider “ensure” and “insure” interchangeable. We prefer ensure when it is not talking about insurance.
',35)]))}const m=t(a,[["render",n]]);export{d as __pageData,m as default};
diff --git a/assets/devdoc_docs_HPCCStyleGuide.md.ttTlrCG0.lean.js b/assets/devdoc_docs_HPCCStyleGuide.md.ttTlrCG0.lean.js
new file mode 100644
index 00000000000..bb353876463
--- /dev/null
+++ b/assets/devdoc_docs_HPCCStyleGuide.md.ttTlrCG0.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as i,a3 as o,o as s}from"./chunks/framework.DkhCEVKm.js";const d=JSON.parse('{"title":"HPCC Writing Style Guide","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/docs/HPCCStyleGuide.md","filePath":"devdoc/docs/HPCCStyleGuide.md","lastUpdated":1731340314000}'),a={name:"devdoc/docs/HPCCStyleGuide.md"};function n(r,e,l,p,u,h){return s(),i("div",null,e[0]||(e[0]=[o('
We strive to maintain a consistent voice. These guidelines can help your writing match that voice.
Use present tense, active voice. Documentation should be you speaking directly to the reader. Simply tell them what to do.
This example sentence: "The user selects the file menu." is passive voice. You wouldn’t say it that way in a conversation.
You should use active voice wording, such as: "Select the file menu".
Similarly, instructions like these are active voice: "Press the button" or "Submit the file".
Documentation is you instructing the user. Just tell them what to do.
Be Brief and keep it simple. Be efficient with your words. Keep sentences short and concise. Keep paragraphs short as well. Use just a few sentences per paragraph. Use simple words wherever possible and try to avoid lengthy explanations.
Consistency: Be consistent. Use the same voice across all documents. Use the same term, use the same spelling, punctuation, etc. Follow the conventions in this guide.
When writing formal documentation that is more than a new feature announcement, do not refer to a new feature, or coming features as such, this does not hold up well over time.
Do not refer to how things were done in the past. For example, "in the past we had to do X-Y-Z steps and now we no longer have to. This ‘new feature’ can do it in one step, Z". This only adds potential confusion. Just instruct on exactly what needs to be done now, using words efficiently as possible, so for this example just say “perform step Z”
There are many terms specific to HPCC Systems® and the HPCC Systems platform. Use the following style guide for word usage and capitalization guidelines to use when referring to system components or other HPCC-specific entities. Maintain consistent usage throughout all docs.
Officially and legally the organizaion's name is HPCC Systems® and it is a registered trademark.
You should always refer to the platform as the HPCC Systems® platform and the registration mark ® should appear in the first and most prominent mention of the name.
While it is acceptable to use the ® anywhere in a document, it is required to be used in the first and most prominent mention - so the average reader will be aware. Any usage after that first and most prominent is optional.
Components and Tools
HPCC Systems Platform
Dali
Sasha
Thor
hThor
ROXIE
DFU Server (Distributed File Utility Server)
ESP Server (Enterprise Server Platform)
ESP Services
WsECL
ECL Watch
ECL Server
ECLCC Server
ECL Agent
ECL IDE
ECL Plug-in for Eclipse
ECL Playground
LDAP
dafilesrv
VS Code (no hyphen)
ECL Language Extension for VS Code
Configuration Manager (not ConfigMgr or ConfigManager)
Note: when referring to the startup command, use configmgr (always lowercase)
To “assure” a person of something is to make him or her confident of it.
According to the Associated Press style, to “ensure” that something happens is to make certain that it does, and to “insure” is to issue an insurance policy.
Other authorities, however, consider “ensure” and “insure” interchangeable. We prefer ensure when it is not talking about insurance.
',35)]))}const m=t(a,[["render",n]]);export{d as __pageData,m as default};
diff --git a/assets/devdoc_newActivity.md.Cwh2eRu1.js b/assets/devdoc_newActivity.md.Cwh2eRu1.js
new file mode 100644
index 00000000000..f3621115ec6
--- /dev/null
+++ b/assets/devdoc_newActivity.md.Cwh2eRu1.js
@@ -0,0 +1,11 @@
+import{_ as t,c as i,a3 as a,o}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Quantile 1 - What is it?","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/newActivity.md","filePath":"devdoc/newActivity.md","lastUpdated":1731340314000}'),s={name:"devdoc/newActivity.md"};function n(r,e,l,h,d,c){return o(),i("div",null,e[0]||(e[0]=[a(`
This series of blog posts started life as a series of walk-throughs and brainstorming sessions at a team offsite. This series will look at adding a new activity to the system. The idea is to give an walk through of the work involved, to highlight the different areas that need changing, and hopefully encourage others to add their own activities. In parallel with the description in this blog there is a series of commits to the github repository that correspond to the different stages in adding the activity. Once the blog is completed, the text will also be checked into the source control tree for future reference.
The new activity is going to be a QUANTILE activity, which can be used to find the records that split a dataset into equal sized blocks. Two common uses are to find the median of a set of data (split into 2) or percentiles (split into 100). It can also be used to split a dataset for distribution across the nodes in a system. One hope is that the classes used to implement quantile in Thor can also be used to improve the performance of the global sort operation.
It may seem fatuous, but the first task in adding any activity to the system is to work out what that activity is going to do! You can approach this in an iterative manner - starting with a minimal set of functionality and adding options as you think of them - or start with a more complete initial design. We have used both approaches in the past to add capabilities to the HPCC system, but on this occasion we will be starting from a more complete design - the conclusion of our initial design discussion:
"What are the inputs, options and capabilities that might be useful in a QUANTILE activity?"
The discussion produced the following items:
Which dataset is being processed?
This is always required and should be the first argument to the activity.
How many parts to split the dataset into?
This is always required, so it should be the next argument to the activity.
Which fields are being used to order (and split) the dataset?
Again this is always required, so the list of fields should follow the number of partitions.
Which fields are returned?
Normally the input row, but often it would be useful for the output to include details of which quantile a row corresponds to. To allow this an optional transform could be passed the input row as LEFT and the quantile number as COUNTER.
How about first and last rows in the dataset?
Sometimes it is also useful to know the first and last rows. Add flags to allow them to be optionally returned.
How do you cope with too few input rows (including an empty input)?
After some discussion we decided that QUANTILE should always return the number of parts requested. If there were fewer items in the input they would be duplicated as appropriate. We should provide a DEDUP flag for the situations when that is not desired. If there is an empty dataset as input then the default (blank) row will be created.
Should all rows have the same weighting?
Generally you want the same weighting for each row. However, if you are using QUANTILE to split your dataset, and the cost of the next operation depends on some feature of the row (e.g., the frequency of the firstname) then you may want to weight the rows differently.
What if we are only interested in the 5th and 95th centiles?
We could optionally allow a set of values to be selected from the results.
There were also some implementation details concluded from the discussions:
How accurate should the results be?
The simplest implementation of QUANTILE (sort and then select the correct rows) will always produce accurate results. However, there may be some implementations that can produce an approximate answer more quickly. Therefore we could add a SKEW attribute to allow early termination.
Does the implementation need to be stable?
In other words, if there are rows with identical values for the ordering fields, but other fields not part of the ordering with different values, does it matter which of those rows are returned? Does the relative order within those matching rows matter?
The general principle in the HPCC system is that sort operations should be stable, and that where possible activities return consistent, reproducible results. However, that often has a cost - either in performance or memory consumption. The design discussion highlighted the fact that if all the fields from the row are included in the sort order then the relative order does not matter because the duplicate rows will be indistinguishable. (This is also true for sorts, and following the discussion an optimization was added to 5.2 to take advantage of this.) For the QUANTILE activity we will add an ECL flag, but the code generator should also aim to spot this automatically.
Returning counts of the numbers in each quantile might be interesting.
This has little value when the results are exact, but may be more useful when a SKEW is specified to allow an approximate answer, or if a dataset might have a vast numbers of duplicates. It is possibly something to add to a future version of the activity. For an approximate answer, calculating the counts is likely to add an additional cost to the implementation, so the target engine should be informed if this is required.
Is the output always sorted by the partition fields?
If this naturally falls out of the implementations then it would be worth including it in the specification. Initially we will assume not, but will revisit after it has been implemented.
After all the discussions we arrived at the following syntax:
QUANTILE(<dataset>, <number-of-ranges>, { sort-order } [, <transform>(LEFT, COUNTER)]
+ [,FIRST][,LAST][,SKEW(<n>)][,UNSTABLE][,SCORE(<score>)][,RANGE(set)][,DEDUP][,LOCAL]
+
+FIRST - Match the first row in the input dataset (as quantile 0)
+LAST - Match the last row in the input dataset (as quantile <n>)
+SKEW - The maximum deviation from the correct results allowed. Defaults to 0.
+UNSTABLE - Is the order of the original input values unimportant?
+SCORE - What weighting should be applied for each row. Defaults to 1.
+RANGE - Which quantiles should actually be returned. (Defaults to ALL).
+DEDUP - Avoid returning a match for an input row more than once.
+
We also summarised a few implementation details:
The activity needs to be available in GLOBAL, LOCAL and GROUPED variants.
The code generator should derive UNSTABLE if no non-sort fields are returned.
Flags to indicate if a score/range is required.
Flag to indicate if a transform is required.
Finally, deciding on the name of the activity took almost as long as designing it!
The end result of this process was summarised in a JIRA issue: https://track.hpccsystems.com/browse/HPCC-12267, which contains details of the desired syntax and semantics. It also contains some details of the next blog topic - test cases.
Incidentally, a question that arose from of the design discussion was "What ECL can we use if we want to annotate a dataset with partition points?". Ideally the user needs a join activity which walks through a table of rows, and matches against the first row that contains key values less than or equal to the values in the search row. There are other situations where that operation would also be useful. Our conclusion was that the system does not have a simple way to achieve that, and that it was a deficiency in the current system, so another JIRA was created (see https://track.hpccsystems.com/browse/HPCC-13016). This is often how the design discussions proceed, with discussions in one area leading to new ideas in another. Similarly we concluded it would be useful to distribute rows in a dataset based on a partition (see https://track.hpccsystems.com/browse/HPCC-13260).
When adding new features to the system, or changing the code generator, the first step is often to write some ECL test cases. They have proved very useful for several reasons:
Developing the test cases can help clarify issues, and other details that the implementation needs to take into account. (E.g., what happens if the input dataset is empty?)
They provide something concrete to aim towards when implementing the feature.
They provide a set of milestones to show progress.
They can be used to check the implementation on the different engines.
As part of the design discussion we also started to create a list of useful test cases (they follow below in the order they were discussed). The tests perform varying functions. Some of the tests are checking that the core functionality works correctly, while others check unusual situations and that strange boundary cases are covered. The tests are not exhaustive, but they are a good starting point and new tests can be added as the implementation progresses.
The following is the list of tests that should be created as part of implementing this activity:
Compare with values extracted from a SORT. Useful to check the implementation, but also to ensure we clearly define which results we are expecting.
QUANTILE with a number-of-ranges = 1, 0, and a very large number. Should also test the number of ranges can be dynamic as well as a constant.
Empty dataset as input.
All input entries are duplicates.
Dataset smaller than number of ranges.
Input sorted and reverse sorted.
Normal data with small number of entries.
Duplicates in the input dataset that cause empty ranges.
Random distribution of numbers without duplicates.
Local and grouped cases.
SKEW that fails.
Test scoring functions.
Testing different skews that work on the same dataset.
An example that uses all the keywords.
Examples that do and do not have extra fields not included in the sort order. (Check that the unstable flag is correctly deduced.)
Globally partitioned already (e.g., globally sorted). All partition points on a single node.
Apply quantile to a dataset, and also to the same dataset that has been reordered/distributed. Check the resulting quantiles are the same.
Calculate just the 5 and 95 centiles from a dataset.
Check a non constant number of splits (and also in a child query where it depends on the parent row).
A transform that does something interesting to the sort order. (Check any order is tracked correctly.)
Check the counts are correct for grouped and local operations.
Call in a child query with options that depend on the parent row (e.g., num partitions).
Split points that fall in the middle of two items.
No input rows and DEDUP attribute specified.
Ideally any test cases for features should be included in the runtime regression suite, which is found in the testing/regress directory in the github repository. Tests that check invalid syntax should go in the compiler regression suite (ecl/regress). Commit https://github.com/ghalliday/HPCC-Platform/commit/d75e6b40e3503f851265670a27889d8adc73f645 contains the test cases so far. Note, the test examples in that commit do not yet cover all the cases above. Before the final pull request for the feature is merged the list above should be revisited and the test suite extended to include any missing tests.
In practice it may be easier to write the test cases in parallel with implementing the parser -since that allows you to check their syntax. Some of the examples in the commit were created before work was started on the parser, others during, and some while implementing the feature itself.
The first stage in implementing QUANTILE will be to add it to the parser. This can sometimes highlight issues with the syntax and cause revisions to the design. In this case there were two technical issues integrating the syntax into the grammar. (If you are not interested in shift/reduce conflicts you may want to skip a few paragraphs and jump to the walkthrough of the changes.)
Originally, the optional transform was specified inside an attribute, e.g., something like OUTPUT(transform). However, this was not very consistent with the way that other transforms were implemented, so the syntax was updated so it became an optional transform following the partition field list.
When the syntax was added to the grammar we hit another problem: Currently, a single production (sortList) in the grammar is used for matching sort orders. As well as accepting fields from a dataset the sort order production has been extended to accept any named attribute that can follow a sort order (e.g., LOCAL). This is because (with one token lookahead) it is ambiguous where the sort order finishes and the list of attributes begins. Trying to include transforms in those productions revealed other problems:
If a production has a sortList followed by a transform (or attribute) then it introduces a shift/reduce error on ','. To avoid the ambiguity all trailing attributes or values need to be included in the sortList.
Including a transform production in the sortList elements causes problems with other transform disambiguation (e.g., DATASET[x] and AGGREGATE).
We could require an attribute around the transform e.g., OUTPUT(transform), but that does not really fit in with other activities in the language.
We could change the parameter order, e.g., move the transform earlier, but that would make the syntax counter-intuitive.
We could require { } around the list - but this is inconsistent with some of the other sort orders.
In order to make some progress I elected to choose the last option and require the sort order to be included in curly braces. There are already a couple of activities - subsort and a form of atmost that similarly require them (and if redesigning ECL from scratch I would be tempted to require them everywhere). The final syntax is something that will need further discussion as part of the review of the pull request though, and may need to be revisited.
The ECL query is represented by a graph of "expression" nodes - each has a "kind" that comes from the enumeration _node_operator. The first requirement is to add a new enumeration to represent the new activity - in this case we elected to reuse an unused placeholder. (These placeholders correspond to some old operators that are no longer supported. They have not been removed because the other elements in the enumeration need to keep the same values since they are used for calculating derived persistent values e.g., the hashes for persists.)
New attribute names in hqlatoms.
The quantile activity introduces some new attribute names that have not been used before. All names are represented in an atom table, so the code in hqlatoms.hpp/cpp is updated to define the new atoms.
Properties of no_quantile
There are various places that need to be updated to allow the system to know about the properties of the new operator:
hqlattr
This contains code to calculate derived attributes. The first entry in the case statement is currently unused (the function should be removed). The second, inside calcRowInformation(), is used to predict how many rows are generated by this activity. This information is percolated through the graph and is used for optimizations, and input counts can be used to select the best implementation for a particular activity.
hqlexpr
Most changes are relatively simple including the text for the operator, whether it is constant, and the number of dataset arguments it has. One key function is getChildDatasetType() that indicates the kind of dataset arguments the operator has, which in turn controls how LEFT/RIGHT are interpreted. In this case some of the activity arguments (e.g., the number of quantiles) implicitly use fields within the parent dataset, and the transform uses LEFT, so the operator returns childdataset_datasetleft.
hqlir
This entry is used for generating an intermediate representation of the graph. This can be useful for debugging issues. (Running eclcc with the logging options "--logfile xxx" and "--logdetail 999" will include details of the expression tree at each point in the code generation process in the log file. Also defining -ftraceIR will output the graphs in the IR format.)
hqlfold
This is the constant folder. At the moment the only change is to ensure that fields that are assigned constants within the transform are processed correctly. Future work could add code to optimize quantile applied to an empty dataset, or selecting 1 division.
hqlmeta
Similar to the functions in hqlattr that calculate derived attributes, these functions are used to calculate how the rows coming out of an activity are sorted, grouped and distributed. It is vital to only preserve information that is guaranteed to be true - otherwise invalid optimizations might be performed on the rest of the expression tree.
reservedwords.cpp
A new entry indicating which category the keyword belongs to.
Finally we have the changes to the parser to recognise the new syntax:
hqllex.l
This file contains the lexer that breaks the ecl file into tokens. There are two new tokens - QUANTILE and SCORE.
Hqlgram.y
This file contains the grammar that matches the language. There are two productions - one that matches the version of QUANTILE with a transform and one without. (Two productions are used instead of an optional transform to avoid shift/reduce errors.)
hqlgram2.cpp
This contains the bulk of the code that is executed by the productions in the grammar. Changes here include new entries added to a case statement to get the text for the new tokens, and a new entry in the simplify() call. This helps reduce the number of valid tokens that could follow when reporting a syntax error.
Looking back over those changes, one reflection is that there are lots of different places that need to be changed. How does a programmer know which functions need to change, and what happens if some are missed? In this example, the locations were found by searching for an activity with a similar syntax e.g., no_soapcall_ds or no_normalize.
It is too easy to miss something, especially for somebody new to the code - although if you do then you will trigger a runtime internal error. It would be much better if the code was refactored so that the bulk of the changes were in one place. (See JIRA https://track.hpccsystems.com/browse/HPCC-13434 that has been added to track improvement of the situation.)
With these changes implemented the examples from the previous pull request now syntax check. The next stage in the process involves thinking through the details of how the activity will be implemented.
The next stage in adding a new activity to the system is to define the interface between the generated code and the engines. The important file for this stage is rtl/include/eclhelper.hpp, which contains the interfaces between the engines and the generated code. These interfaces define the information required by the engines to customize each of the different activities. The changes that define the interface for quantile are found in commit https://github.com/ghalliday/HPCC-Platform/commit/06534d8e9962637fe9a5188d1cc4ab32c3925010.
Adding a quantile activity involves the following changes:
ThorActivityKind - TAKquantile
Each activity that the engines support has an entry in this enumeration. This value is stored in the graph as the _kind attribute of the node.
ActivityInterfaceEnum - TAIquantilearg_1
This enumeration in combination with the selectInterface() member of IHThorArg provides a mechanism for helper interfaces to be extended while preserving backwards compatibility with older workunits. The mechanism is rarely used (but valuable when it is), and adding a new activity only requires a single new entry.
IHThorArg
This is the base interface that all activity interfaces are derived from. This interface does not need to change, but it is worth noting because each activity defines a specialized version of it. The names of the specialised interfaces follow a pattern; in this case the new interface is IHThorQuantileArg.
IHThorQuantileArg
The following is an outline of the new member functions, with comments on their use:
getFlags()
Many of the interfaces have a getFlags() function. It provides a concise way of returning several Boolean options in a single call - provided those options do not change during the execution of the activity. The flags are normally defined with explicit values in an enumeration before the interface. The labels often follow the pattern T<First-letter-of-activity>F<lowercase-name>, i.e. TQFxxx ~= Thor-Quantile-Flag-XXX.
getNumDivisions()
Returns how many parts to split the dataset into.
getSkew()
Corresponds to the SKEW() attribute.
queryCompare()
Returns an implementation of the interface used to compare two rows.
createDefault(rowBuilder)
A function used to create a default row - used if there are no input rows.
transform(rowBuilder, _left, _counter)
The function to create the output record from the input record and the partition number (passed as counter).
getScore(_left)
What weighting should be given to this row?
getRange(isAll, tlen, tgt)
Corresponds to the RANGE attribute.
Note that the different engines all use the same specialised interface - it contains a superset of the functions required by the different targets. Occasionally some of the engines do not need to use some of the functions (e.g., to serialize information between nodes) so the code generator may output empty implementations.
For each interface defined in eclhelper.hpp there is a base implementation class defined in eclhelper_base.hpp. The classes generated for each activity in a query by the code generator are derived from one of these base classes. Therefore we need to create a corresponding new class CThorQuantileArg. It often provides default implementations for some of the helper functions to help reduce the size of the generated code (e.g., getScore returning 1).
Often the process of designing the helper interface is dynamic. As the implementation is created, new options or possibilities for optimizations appear. These require extensions and changes to the helper interface in order to be implemented by the engines. Once the initial interface has been agreed, work on the code generator and the engines can proceeded in parallel. (It is equally possible to design this interface before any work on the parser begins, allowing more work to overlap.)
There are some more details on the contents of thorhelper.hpp in the documentation ecl/eclcc/WORKUNIT.rst within the HPCC repository.
Adding a new activity to the code generator is (surprisingly!) a relatively simple operation. The process is more complicated if the activity also requires an implementation that generates inline C++, but that only applies to a small subset of very simple activities, e.g., filter, aggregate. Changes to the code generator also tend to be more substantial if you add a new type, but that is also not the case for the quantile activity.
For quantile, the only change required is to add a function that generates an implementation of the helper class. The code for all the different activities follows a very similar pattern - generate input activities, generate the helper for this activity, and link the input activities to this new activity. It is often easiest to copy the boiler-plate code from a similar activity (e.g., sort) and then adapt it. (Yes, some of this code could also be refactored... any volunteers?) There are a large number of helper functions available to help generate transforms and other member functions, which also simplifies the process.
Most of the code should be self explanatory, but one item is worth highlighting. The code generator builds up a structure in memory that represents the C++ code that is being generated. The BuildCtx class is used to represent a location within that generated code where new code can be inserted. The instance variable contains several BuildCtx members that are used to represent locations to generate code within the helper class (classctx, nestedctx, createctx and startctx). They are used for different purposes:
classctx
Used to generate any member functions that can be called as soon as the helper object has been created, e.g., getFlags().
nestedctx
Used to generate nested member classes and objects - e.g., comparison classes.
startctx
Any function that may return a value that depends on the context/parent activity. For example if QUANTILE is used inside the TRANSFORM of a PROJECT, the number of partition points may depend on a field in the LEFT row of the PROJECT. Therefore the getNumDivisions() member function needs to be generated inside instance->startctx. These functions can only be called by the engine after onCreate() and onStart() have been called to set up the current context.
createctx
Really, this is a historical artefact from many years ago. It was originally used for functions that could be dependent on a global expression, but not a parent row. Almost all such restrictions have since been removed, and those that remain should probably be replaced with either classctx or startctx.
The only other change is to extend the switch statement in common/thorcommon/thorcommon.cpp to add a text description of the activity.
With the code generator outputting all the information we need, we can now implement the activity in one of the engines. (As I mentioned previously, in practice this is often done in parallel with adding it to the code generator.) Roxie and hThor are the best engines to start with because most of their activities run on a single node - so the implementations tend to be less complicated. It is also relatively easy to debug them, by compiling to create a stand-alone executable, and then running that executable inside a debugger. The following description walks-through the roxie changes:
Code is added to ccdquery to process the new TAKquantile activity kind, and create a factory object of the correct type. The implementation of the factory class is relatively simple - it primarily contains a method for creating an instance of the activity class. Some factories create instances of the helper and cache any information that never changes (in this case the value returned by getFlags(), which is a very marginal optimization).
The classes that implement the existing sort algorithms are extended to return the sorted array in a single call. This allows the quicksort variants to be implemented more efficiently.
The class CRoxieServerQuantileActivity contains the code for implementing the quantile activity. It has the following methods:
Constructor
Extracts any information from the helper that does not vary, and initializes all member variables.
start()
This function is called before the graph is executed. It evaluates any helper methods that might vary from execution to execution (e.g., getRange(), numDivisions()), but which do not depend on the current row.
reset()
Called when a graph has finished executing - after an activity has finished processing all its records. It is used to clean up any variables, and restore the activity ready for processing again (e.g., if it is inside a child query).
needsAllocator()
Returns true if this activity creates new rows.
nextInGroup()
The main function in the activity. This function is called by whichever activity is next in the graph to request a new row from the quantile activity. The functions should be designed so they return the next row as quickly as possible, and delay any processing until it is needed. In this case the input is not read and sorted until the first row is requested.
Note, the call to the helper.transform() returns the size of the resulting row, and returns zero if the row should be skipped. The call to finaliseRowClear() after a successful row creation is there to indicate that the row can no longer be modified, and ensures that any child rows will be correctly freed when the row is freed.
The function also contains extra logic to ensure that groups are implemented correctly. The end of a group is marked by returning a single NULL row, the end of the dataset by two contiguous NULL rows. It is important to ensure that a group that has all its output rows skipped doesn't return two NULLs in a row - hence the checks for anyThisGroup.
With those changes in place, the second commit https://github.com/ghalliday/HPCC-Platform/commit/aeaa209092ea1af9660c6908062c1b0b9acff36b adds support for the RANGE, FIRST, and LAST attributes. It also optimizes the cases where the input is already sorted, and the version of QUANTILE which does not include a transform. (If you are looking at the change in github then it is useful to ignore whitespace changes by appending ?w=1 to the URL). The main changes are
Extra helper methods called in start() to obtain the range.
Optimize the situation where the input is known to be sorted by reading the input rows directly into the "sorted" array.
Extra checks to see if this quantile should be included in the output (FIRST,LAST,RANGE,DEDUP)
An optimization to link the incoming row if the transform does not modify it, by testing the TQFneedtransform flag.
`,78)]))}const m=t(s,[["render",n]]);export{u as __pageData,m as default};
diff --git a/assets/devdoc_newActivity.md.Cwh2eRu1.lean.js b/assets/devdoc_newActivity.md.Cwh2eRu1.lean.js
new file mode 100644
index 00000000000..f3621115ec6
--- /dev/null
+++ b/assets/devdoc_newActivity.md.Cwh2eRu1.lean.js
@@ -0,0 +1,11 @@
+import{_ as t,c as i,a3 as a,o}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Quantile 1 - What is it?","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/newActivity.md","filePath":"devdoc/newActivity.md","lastUpdated":1731340314000}'),s={name:"devdoc/newActivity.md"};function n(r,e,l,h,d,c){return o(),i("div",null,e[0]||(e[0]=[a(`
This series of blog posts started life as a series of walk-throughs and brainstorming sessions at a team offsite. This series will look at adding a new activity to the system. The idea is to give an walk through of the work involved, to highlight the different areas that need changing, and hopefully encourage others to add their own activities. In parallel with the description in this blog there is a series of commits to the github repository that correspond to the different stages in adding the activity. Once the blog is completed, the text will also be checked into the source control tree for future reference.
The new activity is going to be a QUANTILE activity, which can be used to find the records that split a dataset into equal sized blocks. Two common uses are to find the median of a set of data (split into 2) or percentiles (split into 100). It can also be used to split a dataset for distribution across the nodes in a system. One hope is that the classes used to implement quantile in Thor can also be used to improve the performance of the global sort operation.
It may seem fatuous, but the first task in adding any activity to the system is to work out what that activity is going to do! You can approach this in an iterative manner - starting with a minimal set of functionality and adding options as you think of them - or start with a more complete initial design. We have used both approaches in the past to add capabilities to the HPCC system, but on this occasion we will be starting from a more complete design - the conclusion of our initial design discussion:
"What are the inputs, options and capabilities that might be useful in a QUANTILE activity?"
The discussion produced the following items:
Which dataset is being processed?
This is always required and should be the first argument to the activity.
How many parts to split the dataset into?
This is always required, so it should be the next argument to the activity.
Which fields are being used to order (and split) the dataset?
Again this is always required, so the list of fields should follow the number of partitions.
Which fields are returned?
Normally the input row, but often it would be useful for the output to include details of which quantile a row corresponds to. To allow this an optional transform could be passed the input row as LEFT and the quantile number as COUNTER.
How about first and last rows in the dataset?
Sometimes it is also useful to know the first and last rows. Add flags to allow them to be optionally returned.
How do you cope with too few input rows (including an empty input)?
After some discussion we decided that QUANTILE should always return the number of parts requested. If there were fewer items in the input they would be duplicated as appropriate. We should provide a DEDUP flag for the situations when that is not desired. If there is an empty dataset as input then the default (blank) row will be created.
Should all rows have the same weighting?
Generally you want the same weighting for each row. However, if you are using QUANTILE to split your dataset, and the cost of the next operation depends on some feature of the row (e.g., the frequency of the firstname) then you may want to weight the rows differently.
What if we are only interested in the 5th and 95th centiles?
We could optionally allow a set of values to be selected from the results.
There were also some implementation details concluded from the discussions:
How accurate should the results be?
The simplest implementation of QUANTILE (sort and then select the correct rows) will always produce accurate results. However, there may be some implementations that can produce an approximate answer more quickly. Therefore we could add a SKEW attribute to allow early termination.
Does the implementation need to be stable?
In other words, if there are rows with identical values for the ordering fields, but other fields not part of the ordering with different values, does it matter which of those rows are returned? Does the relative order within those matching rows matter?
The general principle in the HPCC system is that sort operations should be stable, and that where possible activities return consistent, reproducible results. However, that often has a cost - either in performance or memory consumption. The design discussion highlighted the fact that if all the fields from the row are included in the sort order then the relative order does not matter because the duplicate rows will be indistinguishable. (This is also true for sorts, and following the discussion an optimization was added to 5.2 to take advantage of this.) For the QUANTILE activity we will add an ECL flag, but the code generator should also aim to spot this automatically.
Returning counts of the numbers in each quantile might be interesting.
This has little value when the results are exact, but may be more useful when a SKEW is specified to allow an approximate answer, or if a dataset might have a vast numbers of duplicates. It is possibly something to add to a future version of the activity. For an approximate answer, calculating the counts is likely to add an additional cost to the implementation, so the target engine should be informed if this is required.
Is the output always sorted by the partition fields?
If this naturally falls out of the implementations then it would be worth including it in the specification. Initially we will assume not, but will revisit after it has been implemented.
After all the discussions we arrived at the following syntax:
QUANTILE(<dataset>, <number-of-ranges>, { sort-order } [, <transform>(LEFT, COUNTER)]
+ [,FIRST][,LAST][,SKEW(<n>)][,UNSTABLE][,SCORE(<score>)][,RANGE(set)][,DEDUP][,LOCAL]
+
+FIRST - Match the first row in the input dataset (as quantile 0)
+LAST - Match the last row in the input dataset (as quantile <n>)
+SKEW - The maximum deviation from the correct results allowed. Defaults to 0.
+UNSTABLE - Is the order of the original input values unimportant?
+SCORE - What weighting should be applied for each row. Defaults to 1.
+RANGE - Which quantiles should actually be returned. (Defaults to ALL).
+DEDUP - Avoid returning a match for an input row more than once.
+
We also summarised a few implementation details:
The activity needs to be available in GLOBAL, LOCAL and GROUPED variants.
The code generator should derive UNSTABLE if no non-sort fields are returned.
Flags to indicate if a score/range is required.
Flag to indicate if a transform is required.
Finally, deciding on the name of the activity took almost as long as designing it!
The end result of this process was summarised in a JIRA issue: https://track.hpccsystems.com/browse/HPCC-12267, which contains details of the desired syntax and semantics. It also contains some details of the next blog topic - test cases.
Incidentally, a question that arose from of the design discussion was "What ECL can we use if we want to annotate a dataset with partition points?". Ideally the user needs a join activity which walks through a table of rows, and matches against the first row that contains key values less than or equal to the values in the search row. There are other situations where that operation would also be useful. Our conclusion was that the system does not have a simple way to achieve that, and that it was a deficiency in the current system, so another JIRA was created (see https://track.hpccsystems.com/browse/HPCC-13016). This is often how the design discussions proceed, with discussions in one area leading to new ideas in another. Similarly we concluded it would be useful to distribute rows in a dataset based on a partition (see https://track.hpccsystems.com/browse/HPCC-13260).
When adding new features to the system, or changing the code generator, the first step is often to write some ECL test cases. They have proved very useful for several reasons:
Developing the test cases can help clarify issues, and other details that the implementation needs to take into account. (E.g., what happens if the input dataset is empty?)
They provide something concrete to aim towards when implementing the feature.
They provide a set of milestones to show progress.
They can be used to check the implementation on the different engines.
As part of the design discussion we also started to create a list of useful test cases (they follow below in the order they were discussed). The tests perform varying functions. Some of the tests are checking that the core functionality works correctly, while others check unusual situations and that strange boundary cases are covered. The tests are not exhaustive, but they are a good starting point and new tests can be added as the implementation progresses.
The following is the list of tests that should be created as part of implementing this activity:
Compare with values extracted from a SORT. Useful to check the implementation, but also to ensure we clearly define which results we are expecting.
QUANTILE with a number-of-ranges = 1, 0, and a very large number. Should also test the number of ranges can be dynamic as well as a constant.
Empty dataset as input.
All input entries are duplicates.
Dataset smaller than number of ranges.
Input sorted and reverse sorted.
Normal data with small number of entries.
Duplicates in the input dataset that cause empty ranges.
Random distribution of numbers without duplicates.
Local and grouped cases.
SKEW that fails.
Test scoring functions.
Testing different skews that work on the same dataset.
An example that uses all the keywords.
Examples that do and do not have extra fields not included in the sort order. (Check that the unstable flag is correctly deduced.)
Globally partitioned already (e.g., globally sorted). All partition points on a single node.
Apply quantile to a dataset, and also to the same dataset that has been reordered/distributed. Check the resulting quantiles are the same.
Calculate just the 5 and 95 centiles from a dataset.
Check a non constant number of splits (and also in a child query where it depends on the parent row).
A transform that does something interesting to the sort order. (Check any order is tracked correctly.)
Check the counts are correct for grouped and local operations.
Call in a child query with options that depend on the parent row (e.g., num partitions).
Split points that fall in the middle of two items.
No input rows and DEDUP attribute specified.
Ideally any test cases for features should be included in the runtime regression suite, which is found in the testing/regress directory in the github repository. Tests that check invalid syntax should go in the compiler regression suite (ecl/regress). Commit https://github.com/ghalliday/HPCC-Platform/commit/d75e6b40e3503f851265670a27889d8adc73f645 contains the test cases so far. Note, the test examples in that commit do not yet cover all the cases above. Before the final pull request for the feature is merged the list above should be revisited and the test suite extended to include any missing tests.
In practice it may be easier to write the test cases in parallel with implementing the parser -since that allows you to check their syntax. Some of the examples in the commit were created before work was started on the parser, others during, and some while implementing the feature itself.
The first stage in implementing QUANTILE will be to add it to the parser. This can sometimes highlight issues with the syntax and cause revisions to the design. In this case there were two technical issues integrating the syntax into the grammar. (If you are not interested in shift/reduce conflicts you may want to skip a few paragraphs and jump to the walkthrough of the changes.)
Originally, the optional transform was specified inside an attribute, e.g., something like OUTPUT(transform). However, this was not very consistent with the way that other transforms were implemented, so the syntax was updated so it became an optional transform following the partition field list.
When the syntax was added to the grammar we hit another problem: Currently, a single production (sortList) in the grammar is used for matching sort orders. As well as accepting fields from a dataset the sort order production has been extended to accept any named attribute that can follow a sort order (e.g., LOCAL). This is because (with one token lookahead) it is ambiguous where the sort order finishes and the list of attributes begins. Trying to include transforms in those productions revealed other problems:
If a production has a sortList followed by a transform (or attribute) then it introduces a shift/reduce error on ','. To avoid the ambiguity all trailing attributes or values need to be included in the sortList.
Including a transform production in the sortList elements causes problems with other transform disambiguation (e.g., DATASET[x] and AGGREGATE).
We could require an attribute around the transform e.g., OUTPUT(transform), but that does not really fit in with other activities in the language.
We could change the parameter order, e.g., move the transform earlier, but that would make the syntax counter-intuitive.
We could require { } around the list - but this is inconsistent with some of the other sort orders.
In order to make some progress I elected to choose the last option and require the sort order to be included in curly braces. There are already a couple of activities - subsort and a form of atmost that similarly require them (and if redesigning ECL from scratch I would be tempted to require them everywhere). The final syntax is something that will need further discussion as part of the review of the pull request though, and may need to be revisited.
The ECL query is represented by a graph of "expression" nodes - each has a "kind" that comes from the enumeration _node_operator. The first requirement is to add a new enumeration to represent the new activity - in this case we elected to reuse an unused placeholder. (These placeholders correspond to some old operators that are no longer supported. They have not been removed because the other elements in the enumeration need to keep the same values since they are used for calculating derived persistent values e.g., the hashes for persists.)
New attribute names in hqlatoms.
The quantile activity introduces some new attribute names that have not been used before. All names are represented in an atom table, so the code in hqlatoms.hpp/cpp is updated to define the new atoms.
Properties of no_quantile
There are various places that need to be updated to allow the system to know about the properties of the new operator:
hqlattr
This contains code to calculate derived attributes. The first entry in the case statement is currently unused (the function should be removed). The second, inside calcRowInformation(), is used to predict how many rows are generated by this activity. This information is percolated through the graph and is used for optimizations, and input counts can be used to select the best implementation for a particular activity.
hqlexpr
Most changes are relatively simple including the text for the operator, whether it is constant, and the number of dataset arguments it has. One key function is getChildDatasetType() that indicates the kind of dataset arguments the operator has, which in turn controls how LEFT/RIGHT are interpreted. In this case some of the activity arguments (e.g., the number of quantiles) implicitly use fields within the parent dataset, and the transform uses LEFT, so the operator returns childdataset_datasetleft.
hqlir
This entry is used for generating an intermediate representation of the graph. This can be useful for debugging issues. (Running eclcc with the logging options "--logfile xxx" and "--logdetail 999" will include details of the expression tree at each point in the code generation process in the log file. Also defining -ftraceIR will output the graphs in the IR format.)
hqlfold
This is the constant folder. At the moment the only change is to ensure that fields that are assigned constants within the transform are processed correctly. Future work could add code to optimize quantile applied to an empty dataset, or selecting 1 division.
hqlmeta
Similar to the functions in hqlattr that calculate derived attributes, these functions are used to calculate how the rows coming out of an activity are sorted, grouped and distributed. It is vital to only preserve information that is guaranteed to be true - otherwise invalid optimizations might be performed on the rest of the expression tree.
reservedwords.cpp
A new entry indicating which category the keyword belongs to.
Finally we have the changes to the parser to recognise the new syntax:
hqllex.l
This file contains the lexer that breaks the ecl file into tokens. There are two new tokens - QUANTILE and SCORE.
Hqlgram.y
This file contains the grammar that matches the language. There are two productions - one that matches the version of QUANTILE with a transform and one without. (Two productions are used instead of an optional transform to avoid shift/reduce errors.)
hqlgram2.cpp
This contains the bulk of the code that is executed by the productions in the grammar. Changes here include new entries added to a case statement to get the text for the new tokens, and a new entry in the simplify() call. This helps reduce the number of valid tokens that could follow when reporting a syntax error.
Looking back over those changes, one reflection is that there are lots of different places that need to be changed. How does a programmer know which functions need to change, and what happens if some are missed? In this example, the locations were found by searching for an activity with a similar syntax e.g., no_soapcall_ds or no_normalize.
It is too easy to miss something, especially for somebody new to the code - although if you do then you will trigger a runtime internal error. It would be much better if the code was refactored so that the bulk of the changes were in one place. (See JIRA https://track.hpccsystems.com/browse/HPCC-13434 that has been added to track improvement of the situation.)
With these changes implemented the examples from the previous pull request now syntax check. The next stage in the process involves thinking through the details of how the activity will be implemented.
The next stage in adding a new activity to the system is to define the interface between the generated code and the engines. The important file for this stage is rtl/include/eclhelper.hpp, which contains the interfaces between the engines and the generated code. These interfaces define the information required by the engines to customize each of the different activities. The changes that define the interface for quantile are found in commit https://github.com/ghalliday/HPCC-Platform/commit/06534d8e9962637fe9a5188d1cc4ab32c3925010.
Adding a quantile activity involves the following changes:
ThorActivityKind - TAKquantile
Each activity that the engines support has an entry in this enumeration. This value is stored in the graph as the _kind attribute of the node.
ActivityInterfaceEnum - TAIquantilearg_1
This enumeration in combination with the selectInterface() member of IHThorArg provides a mechanism for helper interfaces to be extended while preserving backwards compatibility with older workunits. The mechanism is rarely used (but valuable when it is), and adding a new activity only requires a single new entry.
IHThorArg
This is the base interface that all activity interfaces are derived from. This interface does not need to change, but it is worth noting because each activity defines a specialized version of it. The names of the specialised interfaces follow a pattern; in this case the new interface is IHThorQuantileArg.
IHThorQuantileArg
The following is an outline of the new member functions, with comments on their use:
getFlags()
Many of the interfaces have a getFlags() function. It provides a concise way of returning several Boolean options in a single call - provided those options do not change during the execution of the activity. The flags are normally defined with explicit values in an enumeration before the interface. The labels often follow the pattern T<First-letter-of-activity>F<lowercase-name>, i.e. TQFxxx ~= Thor-Quantile-Flag-XXX.
getNumDivisions()
Returns how many parts to split the dataset into.
getSkew()
Corresponds to the SKEW() attribute.
queryCompare()
Returns an implementation of the interface used to compare two rows.
createDefault(rowBuilder)
A function used to create a default row - used if there are no input rows.
transform(rowBuilder, _left, _counter)
The function to create the output record from the input record and the partition number (passed as counter).
getScore(_left)
What weighting should be given to this row?
getRange(isAll, tlen, tgt)
Corresponds to the RANGE attribute.
Note that the different engines all use the same specialised interface - it contains a superset of the functions required by the different targets. Occasionally some of the engines do not need to use some of the functions (e.g., to serialize information between nodes) so the code generator may output empty implementations.
For each interface defined in eclhelper.hpp there is a base implementation class defined in eclhelper_base.hpp. The classes generated for each activity in a query by the code generator are derived from one of these base classes. Therefore we need to create a corresponding new class CThorQuantileArg. It often provides default implementations for some of the helper functions to help reduce the size of the generated code (e.g., getScore returning 1).
Often the process of designing the helper interface is dynamic. As the implementation is created, new options or possibilities for optimizations appear. These require extensions and changes to the helper interface in order to be implemented by the engines. Once the initial interface has been agreed, work on the code generator and the engines can proceeded in parallel. (It is equally possible to design this interface before any work on the parser begins, allowing more work to overlap.)
There are some more details on the contents of thorhelper.hpp in the documentation ecl/eclcc/WORKUNIT.rst within the HPCC repository.
Adding a new activity to the code generator is (surprisingly!) a relatively simple operation. The process is more complicated if the activity also requires an implementation that generates inline C++, but that only applies to a small subset of very simple activities, e.g., filter, aggregate. Changes to the code generator also tend to be more substantial if you add a new type, but that is also not the case for the quantile activity.
For quantile, the only change required is to add a function that generates an implementation of the helper class. The code for all the different activities follows a very similar pattern - generate input activities, generate the helper for this activity, and link the input activities to this new activity. It is often easiest to copy the boiler-plate code from a similar activity (e.g., sort) and then adapt it. (Yes, some of this code could also be refactored... any volunteers?) There are a large number of helper functions available to help generate transforms and other member functions, which also simplifies the process.
Most of the code should be self explanatory, but one item is worth highlighting. The code generator builds up a structure in memory that represents the C++ code that is being generated. The BuildCtx class is used to represent a location within that generated code where new code can be inserted. The instance variable contains several BuildCtx members that are used to represent locations to generate code within the helper class (classctx, nestedctx, createctx and startctx). They are used for different purposes:
classctx
Used to generate any member functions that can be called as soon as the helper object has been created, e.g., getFlags().
nestedctx
Used to generate nested member classes and objects - e.g., comparison classes.
startctx
Any function that may return a value that depends on the context/parent activity. For example if QUANTILE is used inside the TRANSFORM of a PROJECT, the number of partition points may depend on a field in the LEFT row of the PROJECT. Therefore the getNumDivisions() member function needs to be generated inside instance->startctx. These functions can only be called by the engine after onCreate() and onStart() have been called to set up the current context.
createctx
Really, this is a historical artefact from many years ago. It was originally used for functions that could be dependent on a global expression, but not a parent row. Almost all such restrictions have since been removed, and those that remain should probably be replaced with either classctx or startctx.
The only other change is to extend the switch statement in common/thorcommon/thorcommon.cpp to add a text description of the activity.
With the code generator outputting all the information we need, we can now implement the activity in one of the engines. (As I mentioned previously, in practice this is often done in parallel with adding it to the code generator.) Roxie and hThor are the best engines to start with because most of their activities run on a single node - so the implementations tend to be less complicated. It is also relatively easy to debug them, by compiling to create a stand-alone executable, and then running that executable inside a debugger. The following description walks-through the roxie changes:
Code is added to ccdquery to process the new TAKquantile activity kind, and create a factory object of the correct type. The implementation of the factory class is relatively simple - it primarily contains a method for creating an instance of the activity class. Some factories create instances of the helper and cache any information that never changes (in this case the value returned by getFlags(), which is a very marginal optimization).
The classes that implement the existing sort algorithms are extended to return the sorted array in a single call. This allows the quicksort variants to be implemented more efficiently.
The class CRoxieServerQuantileActivity contains the code for implementing the quantile activity. It has the following methods:
Constructor
Extracts any information from the helper that does not vary, and initializes all member variables.
start()
This function is called before the graph is executed. It evaluates any helper methods that might vary from execution to execution (e.g., getRange(), numDivisions()), but which do not depend on the current row.
reset()
Called when a graph has finished executing - after an activity has finished processing all its records. It is used to clean up any variables, and restore the activity ready for processing again (e.g., if it is inside a child query).
needsAllocator()
Returns true if this activity creates new rows.
nextInGroup()
The main function in the activity. This function is called by whichever activity is next in the graph to request a new row from the quantile activity. The functions should be designed so they return the next row as quickly as possible, and delay any processing until it is needed. In this case the input is not read and sorted until the first row is requested.
Note, the call to the helper.transform() returns the size of the resulting row, and returns zero if the row should be skipped. The call to finaliseRowClear() after a successful row creation is there to indicate that the row can no longer be modified, and ensures that any child rows will be correctly freed when the row is freed.
The function also contains extra logic to ensure that groups are implemented correctly. The end of a group is marked by returning a single NULL row, the end of the dataset by two contiguous NULL rows. It is important to ensure that a group that has all its output rows skipped doesn't return two NULLs in a row - hence the checks for anyThisGroup.
With those changes in place, the second commit https://github.com/ghalliday/HPCC-Platform/commit/aeaa209092ea1af9660c6908062c1b0b9acff36b adds support for the RANGE, FIRST, and LAST attributes. It also optimizes the cases where the input is already sorted, and the version of QUANTILE which does not include a transform. (If you are looking at the change in github then it is useful to ignore whitespace changes by appending ?w=1 to the URL). The main changes are
Extra helper methods called in start() to obtain the range.
Optimize the situation where the input is known to be sorted by reading the input rows directly into the "sorted" array.
Extra checks to see if this quantile should be included in the output (FIRST,LAST,RANGE,DEDUP)
An optimization to link the incoming row if the transform does not modify it, by testing the TQFneedtransform flag.
`,78)]))}const m=t(s,[["render",n]]);export{u as __pageData,m as default};
diff --git a/assets/devdoc_roxie.md.C6oOX-36.js b/assets/devdoc_roxie.md.C6oOX-36.js
new file mode 100644
index 00000000000..2a6395f74e7
--- /dev/null
+++ b/assets/devdoc_roxie.md.C6oOX-36.js
@@ -0,0 +1,8 @@
+import{_ as t,c as a,a3 as o,o as i}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Everything you ever wanted to know about Roxie","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/roxie.md","filePath":"devdoc/roxie.md","lastUpdated":1731340314000}'),n={name:"devdoc/roxie.md"};function s(r,e,l,h,d,c){return i(),a("div",null,e[0]||(e[0]=[o(`
Because I could. Many of the pieces needed for Roxie were already created for use in other systems – ECL language, code generator, index creation in Thor, etc. Indexes could be used by Moxie, but that relied on monolithic single-part indexes, was single-threaded (forked a process per query) and had limited ability to do any queries beyond simple index lookups. ECL had already proved itself as a way to express more complex queries concisely, and the concept of doing the processing next to the data had been proved in hOle and Thor, so Roxie – using the same concept for online queries using indexes – was a natural extension of that, reusing the existing index creation and code generation, but adding a new run-time engine geared towards pre-deployed queries and sending index lookup requests to the node holding the index data.
The code generator creates a graph (DAG) representing the query, with one node per activity and links representing the inputs and dependencies. There is also a helper class for each activity.
Roxie loads this graph for all published queries, creating a factory for each activity and recording how they are linked. When a query is executed, the factories create the activity instances and link them together. All activities without output activities (known as ‘sinks’) are then executed (often on parallel threads), and will typically result in a value being written to a workunit, to the socket that the query was received in, or to a global “context” area where subsequent parts of the query might read it.
Data is pulled through the activity graph, by any activity that wants a row from its input requesting it. Evaluation is therefore lazy, with data only calculated as needed. However, to reduce latency in some cases activities will prepare results ahead of when they are requested – for example an index read activity will send the request to the agent(s) as soon as it is started rather than waiting for the data to be requested by its downstream activity. This may result in wasted work, and in some cases may result in data coming back from an agent after the requesting query has completed after discovering it didn’t need it after all – this results in the dreaded “NO msg collator found – using default” tracing (not an error but may be indicative of a query that could use some tuning).
Before requesting rows from an input, it should be started, and when no more rows are required it should be stopped. It should be reset before destruction or reuse (for example for the next row in a child query).
Balancing the desire to reduce latency with the desire to avoid wasted work can be tricky. Conditional activities (IF etc) will not start their unused inputs, so that queries can be written that do different index reads depending on the input. There is also the concept of a “delayed start” activity – I would need to look at the code to remind myself of how those are used.
Splitter activities are a bit painful – they may result in arbitrary buffering of the data consumed by one output until another output is ready to request a row. It’s particularly complex when some of the outputs don’t start at all – the splitter needs to keep track of how many of the inputs have been started and stopped (an input that is not going to be used must be stopped, so that splitters know not to keep data for them). Tracking these start/stop/reset calls accurately is very important otherwise you can end up with weird bugs including potential crashes when activities are destroyed. Therefore we report errors if the counts don’t tally properly at the end of a query – but working out where a call was missed is often not trivial. Usually it’s because of an exception thrown from an unexpected place, e.g. midway through starting.
Note that row requests from the activities above a splitter may execute on whichever thread downstream from the splitter happens to need that particular row first.
The splitter code for tracking whether any of the downstream activities still need a row is a bit hairy/inefficient, IIRC. There may be scope to optimize (but I would recommend adding some good unit test cases first!)
When there are multiple agents fulfilling data on a channel, work is shared among them via a hash of the packet header, which is used to determine which agent should work on that packet. However, if it doesn’t start working on it within a short period (either because the node is down, or because it is too busy on other in-flight requests), then another node may take over. The IBYTI messages are used to indicate that a node has started to work on a packet and therefore there is no need for a secondary to take over.
The priority of agents as determined by the packet hash is also used to determine how to proceed if an IBYTI is received after starting to work on a request. If the IBYTI is from a lower priority buddy (sub-channel) then it is ignored, if it’s from a higher priority one then the processing will be abandoned.
When multicast is enabled, the IBYTI is sent on the same multicast channel as the original packet (and care is needed to ignore ones sent by yourself). Otherwise it is sent to all buddy IPs.
Nodes keep track of how often they have had to step in for a supposedly higher priority node, and reduce their wait time before stepping in each time this happens, so if a node has crashed then the buddy nodes will end up taking over without every packet being delayed.
(QUESTION – does this result in the load on the first node after the failed node getting double the load?)
Newer code for cloud systems (where the topology may change dynamically) send the information about the buddy nodes in the packet header rather than assuming all nodes already have a consistent version of that information. This ensures that all agents are using the same assumptions about buddy nodes and their ordering.
An index is basically a big sorted table of the keyed fields, divided into pages, with an index of the last row from each page used to be able to locate pages quickly. The bottom level pages (‘leaves’) may also contain payload fields that do not form part of the lookup but can be returned with it.
Typical usage within LN Risk tends to lean towards one of two cases:
Many keyed fields with a single “ID” field in the payload
A single “ID” field in the key with many “PII” fields in the payload.
There may be some other cases of note too though – e.g. an error code lookup file which heavily, used, or Boolean search logic keys using smart-stepping to implement boolean search conditions.
It is necessary to store the index pages on disk compressed – they are very compressible – but decompression can be expensive. For this reason traditionally we have maintained a cache of decompressed pages in addition to the cache of compressed pages that can be found in the Linux page cache. However, it would be much preferred if we could avoid decompressing as much as possible, ideally to the point where no significant cache of the decompressed pages was needed.
Presently we need to decompress to search, so we’ve been looking at options to compress the pages in such a way that searching can be done using the compressed form. The current design being played with here uses a form of DFA to perform searching/matching on the keyed fields – the DFA data is a compact representation of the data in the keyed fields but is also efficient to use as for searching. For the payload part, we are looking at several options (potentially using more than one of them depending on the exact data) including:
Do not compress (may be appropriate for ID case, for example)
Compress individual rows, using a shared dictionary (perhaps trained on first n rows of the index)
Compress blocks of rows (in particular, rows that have the same key value)
A fast (to decompress) compression algorithm that handles small blocks of data efficiently is needed. Zstd may be one possible candidate.
Preliminary work to enable the above changes involved some code restructuring to make it possible to plug in different compression formats more easily, and to vary the compression format per page.
It’s used in the cloud to ensure that all nodes can know the IP addresses of all agents currently processing requests for a given channel. These addresses can change over time due to pod restarts or scaling events. Nodes report to the topology server periodically, and it responds to them with the current topology state. There may be multiple topology servers running (for redundancy purposes). If so all reports should go to all, and it should not matter which one’s answer is used. (QUESTION – how is the send to all done?)
All IFileIO objects used to read files from Roxie are instantiated as IRoxieLazyFileIO objects, which means:
The underlying file handles can be closed in the background, in order to handle the case where file handles are a limited resource. The maximum (and minimum) number of open files can be configured separately for local versus remote files (sometimes remote connections are a scarcer resource than local, if there are limits at the remote end).
The actual file connected to can be switched out in the background, to handle the case where a file read from a remote location becomes unavailable, and to switch to reading from a local location after a background file copy operation completes.
Original IBYTI implementation allocated a thread (from the pool) to each incoming query packet, but some will block for a period to allow an IBYTI to arrive to avoid unnecessary work. It was done this way for historical reasons - mainly that the addition of the delay was after the initial IBYTI implementation, so that in the very earliest versions there was no priority given to any particular subchannel and all would start processing at the same time if they had capacity to do so.
This implementation does not seem particularly smart - in particular it's typing up worker threads even though they are not actually working, and may result in the throughput of the Roxie agent being reduced. For that reason an alternative implementation (controlled by the NEW_IBYTI flag) was created during the cloud transition which tracks what incoming packets are waiting for IBYTI expiry via a separate queue, and they are only allocated to a worker thread once the IBYTI delay times out.
So far the NEW_IBYTI flag has only been set on containerized systems (simply to avoid rocking the boat on the bare-metal systems), but we may turn on in bare metal too going forward (and if so, the old version of the code can be removed sooner or later).
Sometimes when developing/debugging Roxie features, it's simplest to run a standalone executable. Using server mode may be useful if wanting to debug server/agent traffic messaging.
For example, to test IBYTI behaviour on a single node, use
Roxie (optionally) maintains a list of the most recently accessed file pages (in a circular buffer), and flushes this information periodically to text files that will persist from one run of Roxie to the next. On startup, these files are processed and the relevant pages preloaded into the linux page cache to ensure that the "hot" pages are already available and maximum performance is available immediately once the Roxie is brought online, rather than requiring a replay of a "typical" query set to heat the cache as used to be done. In particular this should allow a node to be "warm" before being added to the cluster when autoscaling.
There are some questions outstanding about how this operates that may require empirical testing to answer. Firstly, how does this interact with volumes mounted via k8s pvc's, and in particular with cloud billing systems that charge per read. Will the reads that are done to warm the cache be done in large chunks, or will they happen one linux page at a time? The code at the Roxie level operates by memory-mapping the file then touching a byte within each linux page that we want to be "warm", but does the linux paging subsystem fetch larger blocks? Do huge pages play a part here?
Secondly, the prewarm is actually done by a child process (ccdcache), but the parent process is blocked while it happens. It would probably make sense to at allow at least some of the other startup operations of the parent process to proceed in parallel. There are two reasons why the cache prewarm is done using a child process. Firstly is to allow there to be a standalone way to prewarm prior to launching a Roxie, which might be useful for automation in some bare-metal systems. Secondly, because there is a possibility of segfaults resulting from the prewarm if the file has changed size since the cache warming was done, it is easier to contain, capture, and recover from such faults in a child process than it would be inside Roxie. However, it would probably be possible to avoid these segfaults (by checking more carefully against file size before trying to warm a page, for example) and then link the code into Roxie while still keeping the code common with the standalone executable version.
Thirdly, need to check that the prewarm is complete before adding a new agent to the topology. This is especially relevant if we make any change to do the prewarm asynchronously.
Fourthly, there are potential race conditions when reading/writing the file containing cache information, since this file may be written by any agent operating on the same channel, at any time.
Fifthly, how is the amount of information tracked decided? It should be at least related to the amount of memory available to the linux page cache, but that's not a completely trivial thing to calculate. Should we restrict to the most recent N when outputting, where N is calculated from, for example /proc/meminfo's Active(file) value? Unfortunately on containerized sytems that reflects the host, but perhaps /sys/fs/cgroup/memory.stat can be used instead?
When deciding how much to track, we can pick an upper limit from the pod's memory limit. This could be read from /sys/fs/cgroup/memory.max though we currently read from the config file instead. We should probably (a) subtract the roxiemem size from that and (b) think about a value that will work on bare-metal and fusion too. However, because we don't dedup the entries in the circular buffer used for tracking hot pages until the info is flushed, the appropriate size is not really the same as the memory size.
We track all reads by page, and before writing also add all pages in the jhtree cache with info about the node type. Note that a hit in the jhtree page cache won't be noted as a read OTHER than via this last-minute add.
This isn't really specific to Roxie, but was originally added for federated Roxie systems...
When a Roxie query (or hthor/thor) makes a SOAPCALL, there is an option to specify a list of target gateway IPs, and failover to the next in the list if the first does not respond in a timely fashion. In order to avoid this "timely fashion" check adding an overhead to every query made when a listed gateway is unavailable, we maintain a "blacklist" of nodes that have been seen to fail, and do not attempt to connect to them. There is a "deblacklister" thread that checks periodically whether it is now possible to connect to a previously-blacklisted gateway, and removes it from the list if so.
There are a number of potential questions and issues with this code:
It would appear that a blacklist is applied even when there is only one gateway listed. In this case, the blacklist may be doing more harm than good? I'm not sure that is true - it is still causing rapid failures in cases that are never going to work...
Even when there is only one gateway listed, a blacklist MIGHT still be useful (you don't really want EVERY query to block trying to connect, if the gateway is down - may prefer a fast failure). Also applies when there are multiple records, all being passed to a gateway, and with an ONFAIL.
Is the blacklist shared between all queries? I'm pretty sure it is NOT shared across Roxie nodes... Looks like it is a global object, shared between all queries and activities. However, connections that were started in parallel will all be reported as failed rather than blacklisted, which can make it look like it is maintained per-activity.
Is it only a failed connect that leads to blacklisting, or does a slow/error response also cause a gateway endpoint to be blacklisted? It's only a failed connect.
When deblacklisting, can we check any condition other than "Successfully connected"? If not, blacklisting for any reason other than "Did not connect" feels like a recipe for problems. We only check for a connection (and correspondingly only blacklist for a failed connection).
Are we ever using the functionality where there are more than one gateway listed? Most of the time a load-balancer is a preferable solution...
The "deblacklister" thread seems to add an escalating delay between attempts. Is this delay ever reset? Is it configurable? Is it appropriate?
There's a thread (in the blacklister's pool) for each blacklisted endpoint. These threads will never go away if the endpoint does not recover...
There's a delay of up to 10 seconds in terminating caused by the deblacklister's connect having to timeout before it notices that we are stopping. Can we close the socket as well as interrupting the semaphore?
Does the "reconnect" attempt from the deblacklister cause any pain for the server it is connecting to? Lots of connect attempts without any data could look like a DoS attack...
Retries/timeout seems to translate to Owned<ISocketConnectWait> scw = nonBlockingConnect(ep, timeoutMS == WAIT_FOREVER ? 60000 : timeoutMS*(retries+1)); I am not sure that is correct (a single attempt to connect with a long timeout doesn't feel like it is the same as multiple attempts with shorter timeouts, for example if there is a load balancer in the mix).
Perhaps an option to not use blacklister would solve the immediate issue?
The blacklister uses an array of endpoints - If there were a lot blacklisted, a hash table would be better
Hints to control behaviour of deblacklister would behave unpredictably if multiple activities connected to the same endpoint with different hints unless we make the blacklist lookup match the hint values too.
Deblacklister should use nonBlockingConnect too.
Should the scope of the blacklist be different? Possible scopes are:
Shared across all queries/activities (current behaviour)
Specific to an activity, but shared across queries (i.e. owned by the activity factory)
Specific to all activities in a deployed query (i.e. owned by the query factory)
Specific to a particular activity instance (i.e. owned by the activity object)
Specific to a particular query instance (i.e. owned by the query object)
Options 2 and 4 above would allow all aspects of the blacklisting behaviour to be specified by options on the SOAPCALL. We could control whether or not the blacklister is to be used at all via a SOAPCALL option with any of the above...
The HPCC Platform includes a rudimentary performance tracing feature using periodic stack capture to generate flame graphs. Roxie supports this in 3 ways:
If expert/@profileStartup is set in roxie config, a flame graph is generated for operations during Roxie startup phase.
If @perf is set on an incoming query, a flame graph is generated for the lifetime of that query's execution, and returned along with the query results
If expert/perftrace is set in roxie config, one-shot roxie queries (e.g. eclagent mode) generate a flame graph (currently just to a text file).
The perf trace operates as follows:
A child process is launched that runs the doperf script. This samples the current stack(s) every 0.2s (configurable) to a series of text files.
When tracing is done, these text files are "folded" via a perl script that notes every unique stack and how many times it was seen, one line per unique stack
This folded stack list is filtered to suppress some stacks that are not very interesting
The filtered folded stack list is passed to another perl script that generates an svg file.
The basic info captured at step 1 (or maybe 2) could also be analysed to give other insights, such as:
A list of "time in function, time in children of function".
An expanded list of callers to __GI___lll_lock_wait and __GI___lll_lock_wake, to help spot contended critsecs.
Unfortunately some info present in the original stack text files is lost in the folded summary - in particular related to the TID that the stack is on. Can we spot lifetimes of threads and/or should we treat "stacks" on different threads as different? Thread pools might render this difficult though. There is an option in stack-collapse-elfutils.pl to include the TID when considering whether stacks match, so perhaps we should just (optionally) use that.
In localAgent mode, the global queueManager object (normally a RoxieUdpSocketQueueManager) is replaced by a RoxieLocalQueueManager. Outbound packets are added directly to target queue, inbound are packed into DataBuffers.
There is also "local optimizations" mode where any index operation reading a one-part file (does the same apply to one-part disk files?) just reads it directly on the server (regardless of localAgent setting). Typically still injected into receiver code though as otherwise handling exception cases, limits etc would all be duplicated/messy. Rows created in localOptimization mode are created directly in the caller's row manager, and are injected in serialized format.
Why are inbound not created directly in the desired destination's allocator and then marked as serialized? Some lifespan issues... are they insurmountable? We do pack into dataBuffers rather than MemoryBuffers, which avoids a need to copy the data before the receiver can use it. Large rows get split and will require copying again, but we could set dataBufferSize to be bigger in localAgent mode to mitigate this somewhat.
What is the lifespan issue? In-flight queries may be abandoned when a server-side query fails, times out, or no longer needs the data. Using DataBuffer does not have this issue as they are attached to the query's memory manager/allocation once read. Or we could bypass the agent queue altogether, but rather more refactoring needed for that (might almost be easier to extent the "local optimization" mode to use multiple threads at that point)
abortPending, replyPending, and abortPendingData methods are unimplemented, which may lead to some inefficiencies?
Requests from server to agents are send via UDP (and have a size limit of 64k as a result). Historically they were sent using multicast to go to all agents on a channel at the same time, but since most cloud providers do not support multicast, there has long been an option to avoid multicast and send explicitly to the agent IPs. In bare metal systems these IPs are known via the topology file, and do not change. In cloud systems the topology server provides the IPs of all agents for a channel.
In cloud systems, the list of IPs that a message was sent to is included in the message header, so that the IBYTI messages can be sent without requiring that all agents/servers have the same topology information at any given moment (they will stay in sync because of topology server, but may be temporarily out of sync when nodes are added/removed, until next time topology info is retrieved). This is controled by the SUBCHANNELS_IN_HEADER define.
Packets back from agents to server go via the udplib message-passing code. This can best be described by looking at the sending and receiving sides separately.
When sending, results are split into individual packets (DataBuffers), each designed to be under 1 MTU in size. Traditionally this meant they were 1k, but they can be set larger (8k is good). They do have to be a power of 2 because of how they are allocated from the roxiemem heap. The sender maintains a set of UdpReceiverEntry objects, one for each server that it is conversing with. Each UdpReceiverEntry maintains multiple queues of data packets waiting to be sent, one queue for each priority. The UdpReceiverEntry maintains a count of how many packets are contained across all its queues in packetsQueued, so that it knows if there is data to send.
The priority levels are: 0: Out Of Band 1: Fast lane 2: Standard
This is designed to allow control information to be sent without getting blocked by data, and high priority queries to avoid being blocked by data going to lower priority ones. The mechanism for deciding what packet to send next is a little odd though - rather than sending all higher-priorty packets before any lower-priority ones, it round robins across the queues sending up to N^2 from queue 0 then up to N from queue 1 then 1 from queue 2, where N is set by the UdpOutQsPriority option, or 1 if not set. This may be a mistake - probably any from queue 0 should be sent first, before round-robining the other queues in this fashion.
UdpReceiverEntry objects are also responsible for maintaining a list of packets that have been sent but receiver has not yet indicated that they have arrived.
If an agent has data ready for a given receiver, it will send a requestToSend to that receiver, and wait for a permitToSend response. Sequence numbers are used to handle situations where these messages get lost. A permitToSend that does not contain the expected sequence number is ignored.
`,85)]))}const m=t(n,[["render",s]]);export{u as __pageData,m as default};
diff --git a/assets/devdoc_roxie.md.C6oOX-36.lean.js b/assets/devdoc_roxie.md.C6oOX-36.lean.js
new file mode 100644
index 00000000000..2a6395f74e7
--- /dev/null
+++ b/assets/devdoc_roxie.md.C6oOX-36.lean.js
@@ -0,0 +1,8 @@
+import{_ as t,c as a,a3 as o,o as i}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Everything you ever wanted to know about Roxie","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/roxie.md","filePath":"devdoc/roxie.md","lastUpdated":1731340314000}'),n={name:"devdoc/roxie.md"};function s(r,e,l,h,d,c){return i(),a("div",null,e[0]||(e[0]=[o(`
Because I could. Many of the pieces needed for Roxie were already created for use in other systems – ECL language, code generator, index creation in Thor, etc. Indexes could be used by Moxie, but that relied on monolithic single-part indexes, was single-threaded (forked a process per query) and had limited ability to do any queries beyond simple index lookups. ECL had already proved itself as a way to express more complex queries concisely, and the concept of doing the processing next to the data had been proved in hOle and Thor, so Roxie – using the same concept for online queries using indexes – was a natural extension of that, reusing the existing index creation and code generation, but adding a new run-time engine geared towards pre-deployed queries and sending index lookup requests to the node holding the index data.
The code generator creates a graph (DAG) representing the query, with one node per activity and links representing the inputs and dependencies. There is also a helper class for each activity.
Roxie loads this graph for all published queries, creating a factory for each activity and recording how they are linked. When a query is executed, the factories create the activity instances and link them together. All activities without output activities (known as ‘sinks’) are then executed (often on parallel threads), and will typically result in a value being written to a workunit, to the socket that the query was received in, or to a global “context” area where subsequent parts of the query might read it.
Data is pulled through the activity graph, by any activity that wants a row from its input requesting it. Evaluation is therefore lazy, with data only calculated as needed. However, to reduce latency in some cases activities will prepare results ahead of when they are requested – for example an index read activity will send the request to the agent(s) as soon as it is started rather than waiting for the data to be requested by its downstream activity. This may result in wasted work, and in some cases may result in data coming back from an agent after the requesting query has completed after discovering it didn’t need it after all – this results in the dreaded “NO msg collator found – using default” tracing (not an error but may be indicative of a query that could use some tuning).
Before requesting rows from an input, it should be started, and when no more rows are required it should be stopped. It should be reset before destruction or reuse (for example for the next row in a child query).
Balancing the desire to reduce latency with the desire to avoid wasted work can be tricky. Conditional activities (IF etc) will not start their unused inputs, so that queries can be written that do different index reads depending on the input. There is also the concept of a “delayed start” activity – I would need to look at the code to remind myself of how those are used.
Splitter activities are a bit painful – they may result in arbitrary buffering of the data consumed by one output until another output is ready to request a row. It’s particularly complex when some of the outputs don’t start at all – the splitter needs to keep track of how many of the inputs have been started and stopped (an input that is not going to be used must be stopped, so that splitters know not to keep data for them). Tracking these start/stop/reset calls accurately is very important otherwise you can end up with weird bugs including potential crashes when activities are destroyed. Therefore we report errors if the counts don’t tally properly at the end of a query – but working out where a call was missed is often not trivial. Usually it’s because of an exception thrown from an unexpected place, e.g. midway through starting.
Note that row requests from the activities above a splitter may execute on whichever thread downstream from the splitter happens to need that particular row first.
The splitter code for tracking whether any of the downstream activities still need a row is a bit hairy/inefficient, IIRC. There may be scope to optimize (but I would recommend adding some good unit test cases first!)
When there are multiple agents fulfilling data on a channel, work is shared among them via a hash of the packet header, which is used to determine which agent should work on that packet. However, if it doesn’t start working on it within a short period (either because the node is down, or because it is too busy on other in-flight requests), then another node may take over. The IBYTI messages are used to indicate that a node has started to work on a packet and therefore there is no need for a secondary to take over.
The priority of agents as determined by the packet hash is also used to determine how to proceed if an IBYTI is received after starting to work on a request. If the IBYTI is from a lower priority buddy (sub-channel) then it is ignored, if it’s from a higher priority one then the processing will be abandoned.
When multicast is enabled, the IBYTI is sent on the same multicast channel as the original packet (and care is needed to ignore ones sent by yourself). Otherwise it is sent to all buddy IPs.
Nodes keep track of how often they have had to step in for a supposedly higher priority node, and reduce their wait time before stepping in each time this happens, so if a node has crashed then the buddy nodes will end up taking over without every packet being delayed.
(QUESTION – does this result in the load on the first node after the failed node getting double the load?)
Newer code for cloud systems (where the topology may change dynamically) send the information about the buddy nodes in the packet header rather than assuming all nodes already have a consistent version of that information. This ensures that all agents are using the same assumptions about buddy nodes and their ordering.
An index is basically a big sorted table of the keyed fields, divided into pages, with an index of the last row from each page used to be able to locate pages quickly. The bottom level pages (‘leaves’) may also contain payload fields that do not form part of the lookup but can be returned with it.
Typical usage within LN Risk tends to lean towards one of two cases:
Many keyed fields with a single “ID” field in the payload
A single “ID” field in the key with many “PII” fields in the payload.
There may be some other cases of note too though – e.g. an error code lookup file which heavily, used, or Boolean search logic keys using smart-stepping to implement boolean search conditions.
It is necessary to store the index pages on disk compressed – they are very compressible – but decompression can be expensive. For this reason traditionally we have maintained a cache of decompressed pages in addition to the cache of compressed pages that can be found in the Linux page cache. However, it would be much preferred if we could avoid decompressing as much as possible, ideally to the point where no significant cache of the decompressed pages was needed.
Presently we need to decompress to search, so we’ve been looking at options to compress the pages in such a way that searching can be done using the compressed form. The current design being played with here uses a form of DFA to perform searching/matching on the keyed fields – the DFA data is a compact representation of the data in the keyed fields but is also efficient to use as for searching. For the payload part, we are looking at several options (potentially using more than one of them depending on the exact data) including:
Do not compress (may be appropriate for ID case, for example)
Compress individual rows, using a shared dictionary (perhaps trained on first n rows of the index)
Compress blocks of rows (in particular, rows that have the same key value)
A fast (to decompress) compression algorithm that handles small blocks of data efficiently is needed. Zstd may be one possible candidate.
Preliminary work to enable the above changes involved some code restructuring to make it possible to plug in different compression formats more easily, and to vary the compression format per page.
It’s used in the cloud to ensure that all nodes can know the IP addresses of all agents currently processing requests for a given channel. These addresses can change over time due to pod restarts or scaling events. Nodes report to the topology server periodically, and it responds to them with the current topology state. There may be multiple topology servers running (for redundancy purposes). If so all reports should go to all, and it should not matter which one’s answer is used. (QUESTION – how is the send to all done?)
All IFileIO objects used to read files from Roxie are instantiated as IRoxieLazyFileIO objects, which means:
The underlying file handles can be closed in the background, in order to handle the case where file handles are a limited resource. The maximum (and minimum) number of open files can be configured separately for local versus remote files (sometimes remote connections are a scarcer resource than local, if there are limits at the remote end).
The actual file connected to can be switched out in the background, to handle the case where a file read from a remote location becomes unavailable, and to switch to reading from a local location after a background file copy operation completes.
Original IBYTI implementation allocated a thread (from the pool) to each incoming query packet, but some will block for a period to allow an IBYTI to arrive to avoid unnecessary work. It was done this way for historical reasons - mainly that the addition of the delay was after the initial IBYTI implementation, so that in the very earliest versions there was no priority given to any particular subchannel and all would start processing at the same time if they had capacity to do so.
This implementation does not seem particularly smart - in particular it's typing up worker threads even though they are not actually working, and may result in the throughput of the Roxie agent being reduced. For that reason an alternative implementation (controlled by the NEW_IBYTI flag) was created during the cloud transition which tracks what incoming packets are waiting for IBYTI expiry via a separate queue, and they are only allocated to a worker thread once the IBYTI delay times out.
So far the NEW_IBYTI flag has only been set on containerized systems (simply to avoid rocking the boat on the bare-metal systems), but we may turn on in bare metal too going forward (and if so, the old version of the code can be removed sooner or later).
Sometimes when developing/debugging Roxie features, it's simplest to run a standalone executable. Using server mode may be useful if wanting to debug server/agent traffic messaging.
For example, to test IBYTI behaviour on a single node, use
Roxie (optionally) maintains a list of the most recently accessed file pages (in a circular buffer), and flushes this information periodically to text files that will persist from one run of Roxie to the next. On startup, these files are processed and the relevant pages preloaded into the linux page cache to ensure that the "hot" pages are already available and maximum performance is available immediately once the Roxie is brought online, rather than requiring a replay of a "typical" query set to heat the cache as used to be done. In particular this should allow a node to be "warm" before being added to the cluster when autoscaling.
There are some questions outstanding about how this operates that may require empirical testing to answer. Firstly, how does this interact with volumes mounted via k8s pvc's, and in particular with cloud billing systems that charge per read. Will the reads that are done to warm the cache be done in large chunks, or will they happen one linux page at a time? The code at the Roxie level operates by memory-mapping the file then touching a byte within each linux page that we want to be "warm", but does the linux paging subsystem fetch larger blocks? Do huge pages play a part here?
Secondly, the prewarm is actually done by a child process (ccdcache), but the parent process is blocked while it happens. It would probably make sense to at allow at least some of the other startup operations of the parent process to proceed in parallel. There are two reasons why the cache prewarm is done using a child process. Firstly is to allow there to be a standalone way to prewarm prior to launching a Roxie, which might be useful for automation in some bare-metal systems. Secondly, because there is a possibility of segfaults resulting from the prewarm if the file has changed size since the cache warming was done, it is easier to contain, capture, and recover from such faults in a child process than it would be inside Roxie. However, it would probably be possible to avoid these segfaults (by checking more carefully against file size before trying to warm a page, for example) and then link the code into Roxie while still keeping the code common with the standalone executable version.
Thirdly, need to check that the prewarm is complete before adding a new agent to the topology. This is especially relevant if we make any change to do the prewarm asynchronously.
Fourthly, there are potential race conditions when reading/writing the file containing cache information, since this file may be written by any agent operating on the same channel, at any time.
Fifthly, how is the amount of information tracked decided? It should be at least related to the amount of memory available to the linux page cache, but that's not a completely trivial thing to calculate. Should we restrict to the most recent N when outputting, where N is calculated from, for example /proc/meminfo's Active(file) value? Unfortunately on containerized sytems that reflects the host, but perhaps /sys/fs/cgroup/memory.stat can be used instead?
When deciding how much to track, we can pick an upper limit from the pod's memory limit. This could be read from /sys/fs/cgroup/memory.max though we currently read from the config file instead. We should probably (a) subtract the roxiemem size from that and (b) think about a value that will work on bare-metal and fusion too. However, because we don't dedup the entries in the circular buffer used for tracking hot pages until the info is flushed, the appropriate size is not really the same as the memory size.
We track all reads by page, and before writing also add all pages in the jhtree cache with info about the node type. Note that a hit in the jhtree page cache won't be noted as a read OTHER than via this last-minute add.
This isn't really specific to Roxie, but was originally added for federated Roxie systems...
When a Roxie query (or hthor/thor) makes a SOAPCALL, there is an option to specify a list of target gateway IPs, and failover to the next in the list if the first does not respond in a timely fashion. In order to avoid this "timely fashion" check adding an overhead to every query made when a listed gateway is unavailable, we maintain a "blacklist" of nodes that have been seen to fail, and do not attempt to connect to them. There is a "deblacklister" thread that checks periodically whether it is now possible to connect to a previously-blacklisted gateway, and removes it from the list if so.
There are a number of potential questions and issues with this code:
It would appear that a blacklist is applied even when there is only one gateway listed. In this case, the blacklist may be doing more harm than good? I'm not sure that is true - it is still causing rapid failures in cases that are never going to work...
Even when there is only one gateway listed, a blacklist MIGHT still be useful (you don't really want EVERY query to block trying to connect, if the gateway is down - may prefer a fast failure). Also applies when there are multiple records, all being passed to a gateway, and with an ONFAIL.
Is the blacklist shared between all queries? I'm pretty sure it is NOT shared across Roxie nodes... Looks like it is a global object, shared between all queries and activities. However, connections that were started in parallel will all be reported as failed rather than blacklisted, which can make it look like it is maintained per-activity.
Is it only a failed connect that leads to blacklisting, or does a slow/error response also cause a gateway endpoint to be blacklisted? It's only a failed connect.
When deblacklisting, can we check any condition other than "Successfully connected"? If not, blacklisting for any reason other than "Did not connect" feels like a recipe for problems. We only check for a connection (and correspondingly only blacklist for a failed connection).
Are we ever using the functionality where there are more than one gateway listed? Most of the time a load-balancer is a preferable solution...
The "deblacklister" thread seems to add an escalating delay between attempts. Is this delay ever reset? Is it configurable? Is it appropriate?
There's a thread (in the blacklister's pool) for each blacklisted endpoint. These threads will never go away if the endpoint does not recover...
There's a delay of up to 10 seconds in terminating caused by the deblacklister's connect having to timeout before it notices that we are stopping. Can we close the socket as well as interrupting the semaphore?
Does the "reconnect" attempt from the deblacklister cause any pain for the server it is connecting to? Lots of connect attempts without any data could look like a DoS attack...
Retries/timeout seems to translate to Owned<ISocketConnectWait> scw = nonBlockingConnect(ep, timeoutMS == WAIT_FOREVER ? 60000 : timeoutMS*(retries+1)); I am not sure that is correct (a single attempt to connect with a long timeout doesn't feel like it is the same as multiple attempts with shorter timeouts, for example if there is a load balancer in the mix).
Perhaps an option to not use blacklister would solve the immediate issue?
The blacklister uses an array of endpoints - If there were a lot blacklisted, a hash table would be better
Hints to control behaviour of deblacklister would behave unpredictably if multiple activities connected to the same endpoint with different hints unless we make the blacklist lookup match the hint values too.
Deblacklister should use nonBlockingConnect too.
Should the scope of the blacklist be different? Possible scopes are:
Shared across all queries/activities (current behaviour)
Specific to an activity, but shared across queries (i.e. owned by the activity factory)
Specific to all activities in a deployed query (i.e. owned by the query factory)
Specific to a particular activity instance (i.e. owned by the activity object)
Specific to a particular query instance (i.e. owned by the query object)
Options 2 and 4 above would allow all aspects of the blacklisting behaviour to be specified by options on the SOAPCALL. We could control whether or not the blacklister is to be used at all via a SOAPCALL option with any of the above...
The HPCC Platform includes a rudimentary performance tracing feature using periodic stack capture to generate flame graphs. Roxie supports this in 3 ways:
If expert/@profileStartup is set in roxie config, a flame graph is generated for operations during Roxie startup phase.
If @perf is set on an incoming query, a flame graph is generated for the lifetime of that query's execution, and returned along with the query results
If expert/perftrace is set in roxie config, one-shot roxie queries (e.g. eclagent mode) generate a flame graph (currently just to a text file).
The perf trace operates as follows:
A child process is launched that runs the doperf script. This samples the current stack(s) every 0.2s (configurable) to a series of text files.
When tracing is done, these text files are "folded" via a perl script that notes every unique stack and how many times it was seen, one line per unique stack
This folded stack list is filtered to suppress some stacks that are not very interesting
The filtered folded stack list is passed to another perl script that generates an svg file.
The basic info captured at step 1 (or maybe 2) could also be analysed to give other insights, such as:
A list of "time in function, time in children of function".
An expanded list of callers to __GI___lll_lock_wait and __GI___lll_lock_wake, to help spot contended critsecs.
Unfortunately some info present in the original stack text files is lost in the folded summary - in particular related to the TID that the stack is on. Can we spot lifetimes of threads and/or should we treat "stacks" on different threads as different? Thread pools might render this difficult though. There is an option in stack-collapse-elfutils.pl to include the TID when considering whether stacks match, so perhaps we should just (optionally) use that.
In localAgent mode, the global queueManager object (normally a RoxieUdpSocketQueueManager) is replaced by a RoxieLocalQueueManager. Outbound packets are added directly to target queue, inbound are packed into DataBuffers.
There is also "local optimizations" mode where any index operation reading a one-part file (does the same apply to one-part disk files?) just reads it directly on the server (regardless of localAgent setting). Typically still injected into receiver code though as otherwise handling exception cases, limits etc would all be duplicated/messy. Rows created in localOptimization mode are created directly in the caller's row manager, and are injected in serialized format.
Why are inbound not created directly in the desired destination's allocator and then marked as serialized? Some lifespan issues... are they insurmountable? We do pack into dataBuffers rather than MemoryBuffers, which avoids a need to copy the data before the receiver can use it. Large rows get split and will require copying again, but we could set dataBufferSize to be bigger in localAgent mode to mitigate this somewhat.
What is the lifespan issue? In-flight queries may be abandoned when a server-side query fails, times out, or no longer needs the data. Using DataBuffer does not have this issue as they are attached to the query's memory manager/allocation once read. Or we could bypass the agent queue altogether, but rather more refactoring needed for that (might almost be easier to extent the "local optimization" mode to use multiple threads at that point)
abortPending, replyPending, and abortPendingData methods are unimplemented, which may lead to some inefficiencies?
Requests from server to agents are send via UDP (and have a size limit of 64k as a result). Historically they were sent using multicast to go to all agents on a channel at the same time, but since most cloud providers do not support multicast, there has long been an option to avoid multicast and send explicitly to the agent IPs. In bare metal systems these IPs are known via the topology file, and do not change. In cloud systems the topology server provides the IPs of all agents for a channel.
In cloud systems, the list of IPs that a message was sent to is included in the message header, so that the IBYTI messages can be sent without requiring that all agents/servers have the same topology information at any given moment (they will stay in sync because of topology server, but may be temporarily out of sync when nodes are added/removed, until next time topology info is retrieved). This is controled by the SUBCHANNELS_IN_HEADER define.
Packets back from agents to server go via the udplib message-passing code. This can best be described by looking at the sending and receiving sides separately.
When sending, results are split into individual packets (DataBuffers), each designed to be under 1 MTU in size. Traditionally this meant they were 1k, but they can be set larger (8k is good). They do have to be a power of 2 because of how they are allocated from the roxiemem heap. The sender maintains a set of UdpReceiverEntry objects, one for each server that it is conversing with. Each UdpReceiverEntry maintains multiple queues of data packets waiting to be sent, one queue for each priority. The UdpReceiverEntry maintains a count of how many packets are contained across all its queues in packetsQueued, so that it knows if there is data to send.
The priority levels are: 0: Out Of Band 1: Fast lane 2: Standard
This is designed to allow control information to be sent without getting blocked by data, and high priority queries to avoid being blocked by data going to lower priority ones. The mechanism for deciding what packet to send next is a little odd though - rather than sending all higher-priorty packets before any lower-priority ones, it round robins across the queues sending up to N^2 from queue 0 then up to N from queue 1 then 1 from queue 2, where N is set by the UdpOutQsPriority option, or 1 if not set. This may be a mistake - probably any from queue 0 should be sent first, before round-robining the other queues in this fashion.
UdpReceiverEntry objects are also responsible for maintaining a list of packets that have been sent but receiver has not yet indicated that they have arrived.
If an agent has data ready for a given receiver, it will send a requestToSend to that receiver, and wait for a permitToSend response. Sequence numbers are used to handle situations where these messages get lost. A permitToSend that does not contain the expected sequence number is ignored.
`,85)]))}const m=t(n,[["render",s]]);export{u as __pageData,m as default};
diff --git a/assets/devdoc_userdoc_AzureTipsTricks.md.BUnTMakG.js b/assets/devdoc_userdoc_AzureTipsTricks.md.BUnTMakG.js
new file mode 100644
index 00000000000..c2403ac2837
--- /dev/null
+++ b/assets/devdoc_userdoc_AzureTipsTricks.md.BUnTMakG.js
@@ -0,0 +1 @@
+import{_ as e,c as r,o as s}from"./chunks/framework.DkhCEVKm.js";const _=JSON.parse('{"title":"","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/AzureTipsTricks.md","filePath":"devdoc/userdoc/AzureTipsTricks.md","lastUpdated":1731340314000}'),t={name:"devdoc/userdoc/AzureTipsTricks.md"};function c(o,a,d,i,p,n){return s(),r("div")}const m=e(t,[["render",c]]);export{_ as __pageData,m as default};
diff --git a/assets/devdoc_userdoc_AzureTipsTricks.md.BUnTMakG.lean.js b/assets/devdoc_userdoc_AzureTipsTricks.md.BUnTMakG.lean.js
new file mode 100644
index 00000000000..c2403ac2837
--- /dev/null
+++ b/assets/devdoc_userdoc_AzureTipsTricks.md.BUnTMakG.lean.js
@@ -0,0 +1 @@
+import{_ as e,c as r,o as s}from"./chunks/framework.DkhCEVKm.js";const _=JSON.parse('{"title":"","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/AzureTipsTricks.md","filePath":"devdoc/userdoc/AzureTipsTricks.md","lastUpdated":1731340314000}'),t={name:"devdoc/userdoc/AzureTipsTricks.md"};function c(o,a,d,i,p,n){return s(),r("div")}const m=e(t,[["render",c]]);export{_ as __pageData,m as default};
diff --git a/assets/devdoc_userdoc_Blogs.md.D5ZHWDBF.js b/assets/devdoc_userdoc_Blogs.md.D5ZHWDBF.js
new file mode 100644
index 00000000000..9b3bf6dd497
--- /dev/null
+++ b/assets/devdoc_userdoc_Blogs.md.D5ZHWDBF.js
@@ -0,0 +1 @@
+import{_ as e,c as o,o as t}from"./chunks/framework.DkhCEVKm.js";const i=JSON.parse('{"title":"","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/Blogs.md","filePath":"devdoc/userdoc/Blogs.md","lastUpdated":1731340314000}'),s={name:"devdoc/userdoc/Blogs.md"};function a(c,d,r,n,l,p){return t(),o("div")}const m=e(s,[["render",a]]);export{i as __pageData,m as default};
diff --git a/assets/devdoc_userdoc_Blogs.md.D5ZHWDBF.lean.js b/assets/devdoc_userdoc_Blogs.md.D5ZHWDBF.lean.js
new file mode 100644
index 00000000000..9b3bf6dd497
--- /dev/null
+++ b/assets/devdoc_userdoc_Blogs.md.D5ZHWDBF.lean.js
@@ -0,0 +1 @@
+import{_ as e,c as o,o as t}from"./chunks/framework.DkhCEVKm.js";const i=JSON.parse('{"title":"","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/Blogs.md","filePath":"devdoc/userdoc/Blogs.md","lastUpdated":1731340314000}'),s={name:"devdoc/userdoc/Blogs.md"};function a(c,d,r,n,l,p){return t(),o("div")}const m=e(s,[["render",a]]);export{i as __pageData,m as default};
diff --git a/assets/devdoc_userdoc_README.md.6WX0TGeB.js b/assets/devdoc_userdoc_README.md.6WX0TGeB.js
new file mode 100644
index 00000000000..4a77359f86d
--- /dev/null
+++ b/assets/devdoc_userdoc_README.md.6WX0TGeB.js
@@ -0,0 +1 @@
+import{_ as t,c as o,a3 as r,o as a}from"./chunks/framework.DkhCEVKm.js";const m=JSON.parse('{"title":"User Documentation","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/README.md","filePath":"devdoc/userdoc/README.md","lastUpdated":1731340314000}'),i={name:"devdoc/userdoc/README.md"};function s(n,e,l,c,u,d){return a(),o("div",null,e[0]||(e[0]=[r('
Documentation in this directory is targeted to end-users of the HPCC Systems Platform. This is less-formal documentation intended to be produced and released more quickly than the published HPCC documentation. See HPCC documentation if you would like to contribute to our official docs.
',8)]))}const f=t(i,[["render",s]]);export{m as __pageData,f as default};
diff --git a/assets/devdoc_userdoc_README.md.6WX0TGeB.lean.js b/assets/devdoc_userdoc_README.md.6WX0TGeB.lean.js
new file mode 100644
index 00000000000..4a77359f86d
--- /dev/null
+++ b/assets/devdoc_userdoc_README.md.6WX0TGeB.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as o,a3 as r,o as a}from"./chunks/framework.DkhCEVKm.js";const m=JSON.parse('{"title":"User Documentation","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/README.md","filePath":"devdoc/userdoc/README.md","lastUpdated":1731340314000}'),i={name:"devdoc/userdoc/README.md"};function s(n,e,l,c,u,d){return a(),o("div",null,e[0]||(e[0]=[r('
Documentation in this directory is targeted to end-users of the HPCC Systems Platform. This is less-formal documentation intended to be produced and released more quickly than the published HPCC documentation. See HPCC documentation if you would like to contribute to our official docs.
',8)]))}const f=t(i,[["render",s]]);export{m as __pageData,f as default};
diff --git a/assets/devdoc_userdoc_WikiGuidelines.md.BcZTSYE9.js b/assets/devdoc_userdoc_WikiGuidelines.md.BcZTSYE9.js
new file mode 100644
index 00000000000..5078df0b683
--- /dev/null
+++ b/assets/devdoc_userdoc_WikiGuidelines.md.BcZTSYE9.js
@@ -0,0 +1 @@
+import{_ as e,c as t,o as i}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/WikiGuidelines.md","filePath":"devdoc/userdoc/WikiGuidelines.md","lastUpdated":1731340314000}'),d={name:"devdoc/userdoc/WikiGuidelines.md"};function s(o,a,c,r,n,l){return i(),t("div")}const _=e(d,[["render",s]]);export{u as __pageData,_ as default};
diff --git a/assets/devdoc_userdoc_WikiGuidelines.md.BcZTSYE9.lean.js b/assets/devdoc_userdoc_WikiGuidelines.md.BcZTSYE9.lean.js
new file mode 100644
index 00000000000..5078df0b683
--- /dev/null
+++ b/assets/devdoc_userdoc_WikiGuidelines.md.BcZTSYE9.lean.js
@@ -0,0 +1 @@
+import{_ as e,c as t,o as i}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/WikiGuidelines.md","filePath":"devdoc/userdoc/WikiGuidelines.md","lastUpdated":1731340314000}'),d={name:"devdoc/userdoc/WikiGuidelines.md"};function s(o,a,c,r,n,l){return i(),t("div")}const _=e(d,[["render",s]]);export{u as __pageData,_ as default};
diff --git a/assets/devdoc_userdoc_azure_TipsAndTricks.md.2OyOG7Og.js b/assets/devdoc_userdoc_azure_TipsAndTricks.md.2OyOG7Og.js
new file mode 100644
index 00000000000..6d8e796e2af
--- /dev/null
+++ b/assets/devdoc_userdoc_azure_TipsAndTricks.md.2OyOG7Og.js
@@ -0,0 +1,54 @@
+import{_ as n,c as a,a3 as p,o as e}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Azure Portal FAQs","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/azure/TipsAndTricks.md","filePath":"devdoc/userdoc/azure/TipsAndTricks.md","lastUpdated":1731340314000}'),l={name:"devdoc/userdoc/azure/TipsAndTricks.md"};function t(o,s,i,c,r,d){return e(),a("div",null,s[0]||(s[0]=[p(`
1. go to the resource group service page (https://portal.azure.com/#view/HubsExtension/BrowseResourceGroups)
+2. "Add filter"
+3. Filter on "Admin"
+4. Set the Value to "ALL" or select the names you are interested in (the names in the list are the only ones that have the Admin tag set) This works for all other fields that you can filter on
How do I create a dashboard in Azure?
Answer:
Login to the Azure portal.
+
+Click on the \`hamburger icon\`, located at the top left corner of the page.
+
+Click on \`Dashboard\` to go to your dashboards.
+
+Click on \`Create\`, located at the top left corner.
+
+Click on the \`Custom\` tile.
+
+Edit the input box to name your dashboard.
+
+Click on \`Resource groups\` in the tile gallery.
+
+Click \`Add\`.
+
+Click and drag the \`lower right corner\` of the tile to resize it to your likings.
+
+Click \`Save\` to save your settings.
+
+You should now be taken to your new dashboard.
+
+Click on your new dashboard tile.
+
+Click on \`Add filter\`, located at the top center of the page.
+
+Click on the \`Filter\` input box to reveal the tags.
+
+Select the \`Admin\` tag.
+
+Click on the \`Value\` input box.
+
+Click on \`Select all\` to unselect all.
+
+Select your name.
+
+Click on \`Apply\`.
+
+Next, click on \`Manage view\`, located at the top left of the page.
+
+Select \`Save view\`.
+
+Enter a name for the view in the input box.
+
+Click \`Save\`
+
+
+
+
+
+Click on \`Manage View\`, located at the top left of the page
`,7)]))}const g=n(l,[["render",t]]);export{u as __pageData,g as default};
diff --git a/assets/devdoc_userdoc_azure_TipsAndTricks.md.2OyOG7Og.lean.js b/assets/devdoc_userdoc_azure_TipsAndTricks.md.2OyOG7Og.lean.js
new file mode 100644
index 00000000000..6d8e796e2af
--- /dev/null
+++ b/assets/devdoc_userdoc_azure_TipsAndTricks.md.2OyOG7Og.lean.js
@@ -0,0 +1,54 @@
+import{_ as n,c as a,a3 as p,o as e}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Azure Portal FAQs","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/azure/TipsAndTricks.md","filePath":"devdoc/userdoc/azure/TipsAndTricks.md","lastUpdated":1731340314000}'),l={name:"devdoc/userdoc/azure/TipsAndTricks.md"};function t(o,s,i,c,r,d){return e(),a("div",null,s[0]||(s[0]=[p(`
1. go to the resource group service page (https://portal.azure.com/#view/HubsExtension/BrowseResourceGroups)
+2. "Add filter"
+3. Filter on "Admin"
+4. Set the Value to "ALL" or select the names you are interested in (the names in the list are the only ones that have the Admin tag set) This works for all other fields that you can filter on
How do I create a dashboard in Azure?
Answer:
Login to the Azure portal.
+
+Click on the \`hamburger icon\`, located at the top left corner of the page.
+
+Click on \`Dashboard\` to go to your dashboards.
+
+Click on \`Create\`, located at the top left corner.
+
+Click on the \`Custom\` tile.
+
+Edit the input box to name your dashboard.
+
+Click on \`Resource groups\` in the tile gallery.
+
+Click \`Add\`.
+
+Click and drag the \`lower right corner\` of the tile to resize it to your likings.
+
+Click \`Save\` to save your settings.
+
+You should now be taken to your new dashboard.
+
+Click on your new dashboard tile.
+
+Click on \`Add filter\`, located at the top center of the page.
+
+Click on the \`Filter\` input box to reveal the tags.
+
+Select the \`Admin\` tag.
+
+Click on the \`Value\` input box.
+
+Click on \`Select all\` to unselect all.
+
+Select your name.
+
+Click on \`Apply\`.
+
+Next, click on \`Manage view\`, located at the top left of the page.
+
+Select \`Save view\`.
+
+Enter a name for the view in the input box.
+
+Click \`Save\`
+
+
+
+
+
+Click on \`Manage View\`, located at the top left of the page
`,7)]))}const g=n(l,[["render",t]]);export{u as __pageData,g as default};
diff --git a/assets/devdoc_userdoc_copilot_CopilotPromptTips.md._YhzUQ6a.js b/assets/devdoc_userdoc_copilot_CopilotPromptTips.md._YhzUQ6a.js
new file mode 100644
index 00000000000..61b8eed4679
--- /dev/null
+++ b/assets/devdoc_userdoc_copilot_CopilotPromptTips.md._YhzUQ6a.js
@@ -0,0 +1 @@
+import{_ as t,c as i,a3 as o,o as a}from"./chunks/framework.DkhCEVKm.js";const h=JSON.parse('{"title":"Copilot Prompt Tips","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/copilot/CopilotPromptTips.md","filePath":"devdoc/userdoc/copilot/CopilotPromptTips.md","lastUpdated":1731340314000}'),r={name:"devdoc/userdoc/copilot/CopilotPromptTips.md"};function s(n,e,l,p,c,u){return a(),i("div",null,e[0]||(e[0]=[o('
Unlock the full potential of GitHub Copilot with these carefully curated prompts designed to streamline your workflow and enhance productivity.
Whether you're summarizing complex texts, brainstorming innovative ideas, or creating detailed guides, these prompts will help you get the most out of Copilot's capabilities.
Dive in and discover how these simple yet powerful prompts can save you time and effort in your daily tasks.
These prompts are more focused and are meant to accomplish a specific purpose. They are included here to spark your imagination.
If you think of others that could be included here to share with the world, please send your ideas to docfeedback@hpccsystems.com.
Write comments for this ECL code in javadoc format
Create an ECL File Definition, record structure, and inline dataset with [N (number)] of records with the following fields [list of fields]
Example: Create an ECL file definition with a record structure and an inline dataset with 20 records for a dataset of animals with the following fields: ReferenceNumber, AnimalName, ClassName, FamilyName, Color, Size, and LifeExpectancy.
Write ECL code to classify records into [categories].
Example: Write ECL code to classify customer feedback into neutral, negative, or positive categories.
How to Avoid AI Hallucinations with Good Prompts
AI hallucinations refer to instances where artificial intelligence systems generate information or responses that are incorrect, misleading, or entirely fabricated.
Hallucinations can occur due to various reasons, such as limitations in the training data, inherent biases, or the AI's attempt to provide an answer even when it lacks sufficient context or knowledge.
Understanding and mitigating AI hallucinations is crucial for ensuring the reliability and accuracy of AI-driven applications.
Creating effective prompts is essential to minimize AI hallucinations. Here are some tips to help you craft prompts that lead to accurate and reliable responses:
Be Specific and Clear: Ambiguous prompts can lead to incorrect or irrelevant answers. Clearly define the task and provide specific instructions.
Example: Instead of asking "What is AI?", ask "Explain the concept of artificial intelligence and its primary applications in healthcare."
Provide Context: Give the AI enough background information to understand the task. This helps in generating more accurate responses.
Example: "Given the following text about climate change, summarize the key points in a bullet list."
Use Constraints: Limit the scope of the response by specifying constraints such as word count, format, or specific details to include.
Example: "List 5 benefits of renewable energy in a bullet list, each point not exceeding 20 words."
Ask for Evidence or Sources: Encourage the AI to provide evidence or cite sources for the information it generates.
Example: "Explain the impact of social media on mental health and provide references to recent studies."
Iterative Refinement: Start with a broad prompt and refine it based on the initial responses to get more accurate results.
Example: Begin with "Describe the process of photosynthesis." If the response is too vague, refine it to "Describe the process of photosynthesis in plants, including the role of chlorophyll and sunlight."
By following these guidelines, you can reduce the likelihood of AI hallucinations and ensure that the responses generated are accurate and useful.
',18)]))}const m=t(r,[["render",s]]);export{h as __pageData,m as default};
diff --git a/assets/devdoc_userdoc_copilot_CopilotPromptTips.md._YhzUQ6a.lean.js b/assets/devdoc_userdoc_copilot_CopilotPromptTips.md._YhzUQ6a.lean.js
new file mode 100644
index 00000000000..61b8eed4679
--- /dev/null
+++ b/assets/devdoc_userdoc_copilot_CopilotPromptTips.md._YhzUQ6a.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as i,a3 as o,o as a}from"./chunks/framework.DkhCEVKm.js";const h=JSON.parse('{"title":"Copilot Prompt Tips","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/copilot/CopilotPromptTips.md","filePath":"devdoc/userdoc/copilot/CopilotPromptTips.md","lastUpdated":1731340314000}'),r={name:"devdoc/userdoc/copilot/CopilotPromptTips.md"};function s(n,e,l,p,c,u){return a(),i("div",null,e[0]||(e[0]=[o('
Unlock the full potential of GitHub Copilot with these carefully curated prompts designed to streamline your workflow and enhance productivity.
Whether you're summarizing complex texts, brainstorming innovative ideas, or creating detailed guides, these prompts will help you get the most out of Copilot's capabilities.
Dive in and discover how these simple yet powerful prompts can save you time and effort in your daily tasks.
These prompts are more focused and are meant to accomplish a specific purpose. They are included here to spark your imagination.
If you think of others that could be included here to share with the world, please send your ideas to docfeedback@hpccsystems.com.
Write comments for this ECL code in javadoc format
Create an ECL File Definition, record structure, and inline dataset with [N (number)] of records with the following fields [list of fields]
Example: Create an ECL file definition with a record structure and an inline dataset with 20 records for a dataset of animals with the following fields: ReferenceNumber, AnimalName, ClassName, FamilyName, Color, Size, and LifeExpectancy.
Write ECL code to classify records into [categories].
Example: Write ECL code to classify customer feedback into neutral, negative, or positive categories.
How to Avoid AI Hallucinations with Good Prompts
AI hallucinations refer to instances where artificial intelligence systems generate information or responses that are incorrect, misleading, or entirely fabricated.
Hallucinations can occur due to various reasons, such as limitations in the training data, inherent biases, or the AI's attempt to provide an answer even when it lacks sufficient context or knowledge.
Understanding and mitigating AI hallucinations is crucial for ensuring the reliability and accuracy of AI-driven applications.
Creating effective prompts is essential to minimize AI hallucinations. Here are some tips to help you craft prompts that lead to accurate and reliable responses:
Be Specific and Clear: Ambiguous prompts can lead to incorrect or irrelevant answers. Clearly define the task and provide specific instructions.
Example: Instead of asking "What is AI?", ask "Explain the concept of artificial intelligence and its primary applications in healthcare."
Provide Context: Give the AI enough background information to understand the task. This helps in generating more accurate responses.
Example: "Given the following text about climate change, summarize the key points in a bullet list."
Use Constraints: Limit the scope of the response by specifying constraints such as word count, format, or specific details to include.
Example: "List 5 benefits of renewable energy in a bullet list, each point not exceeding 20 words."
Ask for Evidence or Sources: Encourage the AI to provide evidence or cite sources for the information it generates.
Example: "Explain the impact of social media on mental health and provide references to recent studies."
Iterative Refinement: Start with a broad prompt and refine it based on the initial responses to get more accurate results.
Example: Begin with "Describe the process of photosynthesis." If the response is too vague, refine it to "Describe the process of photosynthesis in plants, including the role of chlorophyll and sunlight."
By following these guidelines, you can reduce the likelihood of AI hallucinations and ensure that the responses generated are accurate and useful.
',18)]))}const m=t(r,[["render",s]]);export{h as __pageData,m as default};
diff --git a/assets/devdoc_userdoc_roxie_FAQ.md.Cbz7RPiY.js b/assets/devdoc_userdoc_roxie_FAQ.md.Cbz7RPiY.js
new file mode 100644
index 00000000000..6d996e5cfe3
--- /dev/null
+++ b/assets/devdoc_userdoc_roxie_FAQ.md.Cbz7RPiY.js
@@ -0,0 +1,24 @@
+import{_ as a,c as e,a3 as t,o as n}from"./chunks/framework.DkhCEVKm.js";const g=JSON.parse('{"title":"ROXIE FAQs","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/roxie/FAQ.md","filePath":"devdoc/userdoc/roxie/FAQ.md","lastUpdated":1731340314000}'),o={name:"devdoc/userdoc/roxie/FAQ.md"};function p(i,s,l,r,c,d){return n(),e("div",null,s[0]||(s[0]=[t(`
How I can compile a query on a containerized or cloud-based system?
Answer:
Same way as bare metal. Command line, or with the IDE, or from ECL Watch. Just point to the HPCC Systems instance to compile.
+For Example:
+ecl deploy <target> <file>
How do I copy queries from an on-prem cluster to Azure?
Answer:
The copy query command – use the Azure host name or IP address for the target.
+For example:
+ecl queries copy <source_query_path> <target_queryset>
How can I get the IP address for the Azure target cluster?
Answer:
Use the "kubectl get svc" command. Use the external IP address listed for ECL Watch.
+kubectl get svc
Do we have to have use the DNSName or do we need to use the IP address?
Answer:
If you can reach ECL Watch with the DNS Name then it should also work for the command line.
How can I find the ECL Watch or Dali hostname?
Answer:
If you did not set up the containerized instance, then you need to ask your Systems Administrator or whomever set it up..
How do I publish a package file?
Answer:
Same way as bare metal.
+To add a new package file: ecl packagemap add or
+To copy exisitng package file : ecl packagemap copy
How do I check the logs?
Answer:
kubectl log <podname>
+in addition you can use -f (follow) option to tail the logs. Optionally you can also issue the <namespace> parameter.
+For example:
+kbectl log roxie-agent-1-3b12a587b –namespace MyNameSpace
+Optionally, you may have implemented a log-processing solution such as the Elastic Stack (elastic4hpcclogs).
How do I get the data on to Azure?
Answer:
Use the copy query command and copy or add the Packagemap.
+With data copy start in the logs…copy from remote location specified if data doesn’t exist on the local system.
+The remote location is the remote Dali (use the --daliip=<daliIP> parameter to specify the remote Dali)
+You can also use ECL Watch.
How can I start a cloud cluster? (akin to the old Virtual Box image)?
Answer:
Can use Docker Desktop, or Azure or any cloud provider and install the HPCC Systems Cloud native helm
+charts
How can I show the ECL queries that are published to a given Roxie?
Answer:
Can use WUListQueries
+For example:
+https://[eclwatch]:18010/WsWorkunits/WUListQueries.json?ver_=1.86&ClusterName=roxie&CheckAllNodes=0
I set up persistent storage on my containerized HPCC Systems, and now it won't start. Why?
Answer:
One possible reason may be that all of the required storage directories are not present. The directories for ~/
+hpccdata/dalistorage, hpcc-data, debug, queries, sasha, and dropzone are all required to exist or your cluster may not start.
Are there any new methods available to work with queries?
Answer:
Yes. There is a new method available ServiceQuery.
+https://[eclwatch]:18010/WsResources/ServiceQuery?ver_=1.01&
+For example Roxie Queries:
+https://[eclwatch]:18010/WsResources/ServiceQuery?ver_=1.01&Type=roxie
+or WsECL (eclqueries)
+https://[eclwatch]:18010/WsResources/ServiceQuery?ver_=1.01&Type=eclqueries
`,37)]))}const u=a(o,[["render",p]]);export{g as __pageData,u as default};
diff --git a/assets/devdoc_userdoc_roxie_FAQ.md.Cbz7RPiY.lean.js b/assets/devdoc_userdoc_roxie_FAQ.md.Cbz7RPiY.lean.js
new file mode 100644
index 00000000000..6d996e5cfe3
--- /dev/null
+++ b/assets/devdoc_userdoc_roxie_FAQ.md.Cbz7RPiY.lean.js
@@ -0,0 +1,24 @@
+import{_ as a,c as e,a3 as t,o as n}from"./chunks/framework.DkhCEVKm.js";const g=JSON.parse('{"title":"ROXIE FAQs","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/roxie/FAQ.md","filePath":"devdoc/userdoc/roxie/FAQ.md","lastUpdated":1731340314000}'),o={name:"devdoc/userdoc/roxie/FAQ.md"};function p(i,s,l,r,c,d){return n(),e("div",null,s[0]||(s[0]=[t(`
How I can compile a query on a containerized or cloud-based system?
Answer:
Same way as bare metal. Command line, or with the IDE, or from ECL Watch. Just point to the HPCC Systems instance to compile.
+For Example:
+ecl deploy <target> <file>
How do I copy queries from an on-prem cluster to Azure?
Answer:
The copy query command – use the Azure host name or IP address for the target.
+For example:
+ecl queries copy <source_query_path> <target_queryset>
How can I get the IP address for the Azure target cluster?
Answer:
Use the "kubectl get svc" command. Use the external IP address listed for ECL Watch.
+kubectl get svc
Do we have to have use the DNSName or do we need to use the IP address?
Answer:
If you can reach ECL Watch with the DNS Name then it should also work for the command line.
How can I find the ECL Watch or Dali hostname?
Answer:
If you did not set up the containerized instance, then you need to ask your Systems Administrator or whomever set it up..
How do I publish a package file?
Answer:
Same way as bare metal.
+To add a new package file: ecl packagemap add or
+To copy exisitng package file : ecl packagemap copy
How do I check the logs?
Answer:
kubectl log <podname>
+in addition you can use -f (follow) option to tail the logs. Optionally you can also issue the <namespace> parameter.
+For example:
+kbectl log roxie-agent-1-3b12a587b –namespace MyNameSpace
+Optionally, you may have implemented a log-processing solution such as the Elastic Stack (elastic4hpcclogs).
How do I get the data on to Azure?
Answer:
Use the copy query command and copy or add the Packagemap.
+With data copy start in the logs…copy from remote location specified if data doesn’t exist on the local system.
+The remote location is the remote Dali (use the --daliip=<daliIP> parameter to specify the remote Dali)
+You can also use ECL Watch.
How can I start a cloud cluster? (akin to the old Virtual Box image)?
Answer:
Can use Docker Desktop, or Azure or any cloud provider and install the HPCC Systems Cloud native helm
+charts
How can I show the ECL queries that are published to a given Roxie?
Answer:
Can use WUListQueries
+For example:
+https://[eclwatch]:18010/WsWorkunits/WUListQueries.json?ver_=1.86&ClusterName=roxie&CheckAllNodes=0
I set up persistent storage on my containerized HPCC Systems, and now it won't start. Why?
Answer:
One possible reason may be that all of the required storage directories are not present. The directories for ~/
+hpccdata/dalistorage, hpcc-data, debug, queries, sasha, and dropzone are all required to exist or your cluster may not start.
Are there any new methods available to work with queries?
Answer:
Yes. There is a new method available ServiceQuery.
+https://[eclwatch]:18010/WsResources/ServiceQuery?ver_=1.01&
+For example Roxie Queries:
+https://[eclwatch]:18010/WsResources/ServiceQuery?ver_=1.01&Type=roxie
+or WsECL (eclqueries)
+https://[eclwatch]:18010/WsResources/ServiceQuery?ver_=1.01&Type=eclqueries
`,37)]))}const u=a(o,[["render",p]]);export{g as __pageData,u as default};
diff --git a/assets/devdoc_userdoc_troubleshoot_ClientsToolIssues.md.Buqpsc4y.js b/assets/devdoc_userdoc_troubleshoot_ClientsToolIssues.md.Buqpsc4y.js
new file mode 100644
index 00000000000..af4aefe1e69
--- /dev/null
+++ b/assets/devdoc_userdoc_troubleshoot_ClientsToolIssues.md.Buqpsc4y.js
@@ -0,0 +1 @@
+import{_ as t,c as o,a3 as s,o as l}from"./chunks/framework.DkhCEVKm.js";const r="/HPCC-Platform/assets/ECLIDE-WindowsProtectionError.ZJeAxFtk.png",a="/HPCC-Platform/assets/ECLIDE-AppProperties.CzKpIibe.png",C=JSON.parse('{"title":"How to resolve issues installing the ECL IDE / Client tools","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/troubleshoot/ClientsToolIssues.md","filePath":"devdoc/userdoc/troubleshoot/ClientsToolIssues.md","lastUpdated":1731340314000}'),i={name:"devdoc/userdoc/troubleshoot/ClientsToolIssues.md"};function n(c,e,d,h,u,p){return l(),o("div",null,e[0]||(e[0]=[s('
How to resolve issues installing the ECL IDE / Client tools
Some users may experience an issue when trying to install the ECL IDE / client tools version 8.10 and later. If you get an error saying something like, Windows Defender SmartScreen prevented access and there is no option other than "Don't Run":
Go to the folder where you downloaded the file.
Once there go to the app file properties click unblock
',6)]))}const f=t(i,[["render",n]]);export{C as __pageData,f as default};
diff --git a/assets/devdoc_userdoc_troubleshoot_ClientsToolIssues.md.Buqpsc4y.lean.js b/assets/devdoc_userdoc_troubleshoot_ClientsToolIssues.md.Buqpsc4y.lean.js
new file mode 100644
index 00000000000..af4aefe1e69
--- /dev/null
+++ b/assets/devdoc_userdoc_troubleshoot_ClientsToolIssues.md.Buqpsc4y.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as o,a3 as s,o as l}from"./chunks/framework.DkhCEVKm.js";const r="/HPCC-Platform/assets/ECLIDE-WindowsProtectionError.ZJeAxFtk.png",a="/HPCC-Platform/assets/ECLIDE-AppProperties.CzKpIibe.png",C=JSON.parse('{"title":"How to resolve issues installing the ECL IDE / Client tools","description":"","frontmatter":{},"headers":[],"relativePath":"devdoc/userdoc/troubleshoot/ClientsToolIssues.md","filePath":"devdoc/userdoc/troubleshoot/ClientsToolIssues.md","lastUpdated":1731340314000}'),i={name:"devdoc/userdoc/troubleshoot/ClientsToolIssues.md"};function n(c,e,d,h,u,p){return l(),o("div",null,e[0]||(e[0]=[s('
How to resolve issues installing the ECL IDE / Client tools
Some users may experience an issue when trying to install the ECL IDE / client tools version 8.10 and later. If you get an error saying something like, Windows Defender SmartScreen prevented access and there is no option other than "Don't Run":
Go to the folder where you downloaded the file.
Once there go to the app file properties click unblock
',6)]))}const f=t(i,[["render",n]]);export{C as __pageData,f as default};
diff --git a/assets/ecl_ecl-bundle_DOCUMENTATION.md.rUiAyrKM.js b/assets/ecl_ecl-bundle_DOCUMENTATION.md.rUiAyrKM.js
new file mode 100644
index 00000000000..b47e2e24d43
--- /dev/null
+++ b/assets/ecl_ecl-bundle_DOCUMENTATION.md.rUiAyrKM.js
@@ -0,0 +1 @@
+import{_ as t,c as l,a3 as a,o as n}from"./chunks/framework.DkhCEVKm.js";const p=JSON.parse('{"title":"Ecl-bundle source documentation","description":"","frontmatter":{},"headers":[],"relativePath":"ecl/ecl-bundle/DOCUMENTATION.md","filePath":"ecl/ecl-bundle/DOCUMENTATION.md","lastUpdated":1731340314000}'),i={name:"ecl/ecl-bundle/DOCUMENTATION.md"};function o(r,e,s,d,c,u){return n(),l("div",null,e[0]||(e[0]=[a('
The ecl-bundle executable (normally executed from the ecl executable by specifying 'ecl bundle XXX' is designed to manipulate ecl bundle files
these are self-contained parcels of ECL code, packaged into a compressed file like a zip or .tar.gz file, that can be downloaded, installed, removed etc from an ECL installation.
The metadata for ECL bundles is described using an exported module called Bundle within the bundle's source tree - typically, this means that a file called Bundle.ecl will be added to the highest level of the bundle's directory tree. In order to extract the information from the Bundle module, eclcc is run in 'evaluate' mode (using the -Me option) which will parse the bundle module and output the required fields to stdout.
ecl-bundle also executes eclcc (using the --showpaths option) to determine where bundle files are to be located.
In order to make versioning easier, bundle files are not copied directly into the bundles directory. A bundle called "MyBundle" that announces itself as version "x.y.z" will be installed to the directory
$BUNDLEDIR/_versions/MyBundle/x.y.z
A "redirect" file called MyBundle.ecl is then created in $BUNDLEDIR, which redirects any IMPORT MyBundle statement to actually import the currently active version of the bundle in _versions/MyBundle/x.y.z
By rewriting this redirect file, it is possible to switch to using a different version of a bundle without having to uninstall and reinstall.
In a future release, we hope to make it possible to specify that bundle A requires version X of bundle B, while bundle C requires version Y of bundle B. That will require the redirect files to be 'local' to a bundle (and will require that bundle B uses a redirect file to ensure it picks up the local copy of B when making internal calls).
An IBundleInfo represents a specific copy of a bundle, and is created by explicitly parsing a snippet of ECL that imports it, with the ECL include path set to include only the specified bundle.
An IBundleInfoSet represents all the installed versions of a particular named bundle.
An IBundleCollection represents all the bundles on the system.
Every individual subcommand is represented by a class derived (directly or indirectly) from EclCmdCommon. These classes are responsible for command-line parsing, usage text output, and (most importantly) execution of the desired outcomes.
',19)]))}const b=t(i,[["render",o]]);export{p as __pageData,b as default};
diff --git a/assets/ecl_ecl-bundle_DOCUMENTATION.md.rUiAyrKM.lean.js b/assets/ecl_ecl-bundle_DOCUMENTATION.md.rUiAyrKM.lean.js
new file mode 100644
index 00000000000..b47e2e24d43
--- /dev/null
+++ b/assets/ecl_ecl-bundle_DOCUMENTATION.md.rUiAyrKM.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as l,a3 as a,o as n}from"./chunks/framework.DkhCEVKm.js";const p=JSON.parse('{"title":"Ecl-bundle source documentation","description":"","frontmatter":{},"headers":[],"relativePath":"ecl/ecl-bundle/DOCUMENTATION.md","filePath":"ecl/ecl-bundle/DOCUMENTATION.md","lastUpdated":1731340314000}'),i={name:"ecl/ecl-bundle/DOCUMENTATION.md"};function o(r,e,s,d,c,u){return n(),l("div",null,e[0]||(e[0]=[a('
The ecl-bundle executable (normally executed from the ecl executable by specifying 'ecl bundle XXX' is designed to manipulate ecl bundle files
these are self-contained parcels of ECL code, packaged into a compressed file like a zip or .tar.gz file, that can be downloaded, installed, removed etc from an ECL installation.
The metadata for ECL bundles is described using an exported module called Bundle within the bundle's source tree - typically, this means that a file called Bundle.ecl will be added to the highest level of the bundle's directory tree. In order to extract the information from the Bundle module, eclcc is run in 'evaluate' mode (using the -Me option) which will parse the bundle module and output the required fields to stdout.
ecl-bundle also executes eclcc (using the --showpaths option) to determine where bundle files are to be located.
In order to make versioning easier, bundle files are not copied directly into the bundles directory. A bundle called "MyBundle" that announces itself as version "x.y.z" will be installed to the directory
$BUNDLEDIR/_versions/MyBundle/x.y.z
A "redirect" file called MyBundle.ecl is then created in $BUNDLEDIR, which redirects any IMPORT MyBundle statement to actually import the currently active version of the bundle in _versions/MyBundle/x.y.z
By rewriting this redirect file, it is possible to switch to using a different version of a bundle without having to uninstall and reinstall.
In a future release, we hope to make it possible to specify that bundle A requires version X of bundle B, while bundle C requires version Y of bundle B. That will require the redirect files to be 'local' to a bundle (and will require that bundle B uses a redirect file to ensure it picks up the local copy of B when making internal calls).
An IBundleInfo represents a specific copy of a bundle, and is created by explicitly parsing a snippet of ECL that imports it, with the ECL include path set to include only the specified bundle.
An IBundleInfoSet represents all the installed versions of a particular named bundle.
An IBundleCollection represents all the bundles on the system.
Every individual subcommand is represented by a class derived (directly or indirectly) from EclCmdCommon. These classes are responsible for command-line parsing, usage text output, and (most importantly) execution of the desired outcomes.
',19)]))}const b=t(i,[["render",o]]);export{p as __pageData,b as default};
diff --git a/assets/ecllibrary_StyleGuide.md.CjMQ9yWq.js b/assets/ecllibrary_StyleGuide.md.CjMQ9yWq.js
new file mode 100644
index 00000000000..4b6c2a2a4b4
--- /dev/null
+++ b/assets/ecllibrary_StyleGuide.md.CjMQ9yWq.js
@@ -0,0 +1,20 @@
+import{_ as i,c as a,a3 as e,o as n}from"./chunks/framework.DkhCEVKm.js";const o=JSON.parse('{"title":"ECL standard library style guide","description":"","frontmatter":{},"headers":[],"relativePath":"ecllibrary/StyleGuide.md","filePath":"ecllibrary/StyleGuide.md","lastUpdated":1731340314000}'),l={name:"ecllibrary/StyleGuide.md"};function t(h,s,p,r,k,d){return n(),a("div",null,s[0]||(s[0]=[e(`
my_record := RECORD
+ INTEGER4 id;
+ STRING firstname{MAXLENGTH(40)};
+ STRING lastname{MAXLENGTH(50)};
+END;
+
+/**
+ * Returns a dataset of people to treat with caution matching a particular lastname. The
+ * names are maintained in a global database of undesirables.
+ *
+ * @param search_lastname A lastname used as a filter
+ * @return The list of people
+ * @see NoFlyList
+ * @see MorePeopleToAvoid
+ */
+
+EXPORT DodgyCharacters(STRING search_lastname) := FUNCTION
+ raw_ds := DATASET(my_record, 'undesirables', THOR);
+ RETURN raw_ds(last_name = search_lastname);
+END;
Some additional rules for attributes in the library:
Services should be SHARED and EXPORTed via intermediate attributes
All attributes must have at least one matching test. If you're not on the test list you're not coming in.
`,7)]))}const c=i(l,[["render",t]]);export{o as __pageData,c as default};
diff --git a/assets/ecllibrary_StyleGuide.md.CjMQ9yWq.lean.js b/assets/ecllibrary_StyleGuide.md.CjMQ9yWq.lean.js
new file mode 100644
index 00000000000..4b6c2a2a4b4
--- /dev/null
+++ b/assets/ecllibrary_StyleGuide.md.CjMQ9yWq.lean.js
@@ -0,0 +1,20 @@
+import{_ as i,c as a,a3 as e,o as n}from"./chunks/framework.DkhCEVKm.js";const o=JSON.parse('{"title":"ECL standard library style guide","description":"","frontmatter":{},"headers":[],"relativePath":"ecllibrary/StyleGuide.md","filePath":"ecllibrary/StyleGuide.md","lastUpdated":1731340314000}'),l={name:"ecllibrary/StyleGuide.md"};function t(h,s,p,r,k,d){return n(),a("div",null,s[0]||(s[0]=[e(`
my_record := RECORD
+ INTEGER4 id;
+ STRING firstname{MAXLENGTH(40)};
+ STRING lastname{MAXLENGTH(50)};
+END;
+
+/**
+ * Returns a dataset of people to treat with caution matching a particular lastname. The
+ * names are maintained in a global database of undesirables.
+ *
+ * @param search_lastname A lastname used as a filter
+ * @return The list of people
+ * @see NoFlyList
+ * @see MorePeopleToAvoid
+ */
+
+EXPORT DodgyCharacters(STRING search_lastname) := FUNCTION
+ raw_ds := DATASET(my_record, 'undesirables', THOR);
+ RETURN raw_ds(last_name = search_lastname);
+END;
Some additional rules for attributes in the library:
Services should be SHARED and EXPORTed via intermediate attributes
All attributes must have at least one matching test. If you're not on the test list you're not coming in.
// Mount / to ./www directory
+auto ret = svr.set_mount_point("/", "./www");
+if (!ret) {
+ // The specified base directory doesn't exist...
+}
+
+// Mount /public to ./www directory
+ret = svr.set_mount_point("/public", "./www");
+
+// Mount /public to ./www1 and ./www2 directories
+ret = svr.set_mount_point("/public", "./www1"); // 1st order to search
+ret = svr.set_mount_point("/public", "./www2"); // 2nd order to search
+
+// Remove mount /
+ret = svr.remove_mount_point("/");
+
+// Remove mount /public
+ret = svr.remove_mount_point("/public");
cpp
// User defined file extension and MIME type mappings
+svr.set_file_extension_and_mimetype_mapping("cc", "text/x-c");
+svr.set_file_extension_and_mimetype_mapping("cpp", "text/x-c");
+svr.set_file_extension_and_mimetype_mapping("hh", "text/x-h");
The followings are built-in mappings:
Extension
MIME Type
txt
text/plain
html, htm
text/html
css
text/css
jpeg, jpg
image/jpg
png
image/png
gif
image/gif
svg
image/svg+xml
ico
image/x-icon
json
application/json
pdf
application/pdf
js
application/javascript
wasm
application/wasm
xml
application/xml
xhtml
application/xhtml+xml
NOTE: These the static file server methods are not thread safe.
// Send a final status without reading the message body.
+svr.set_expect_100_continue_handler([](const Request &req, Response &res) {
+ return res.status = 401;
+});
httplib::Client cli("httpbin.org");
+
+auto res = cli.Get("/range/32", {
+ httplib::make_range_header({{1, 10}}) // 'Range: bytes=1-10'
+});
+// res->status should be 206.
+// res->body should be "bcdefghijk".
Brotli compression is available with CPPHTTPLIB_BROTLI_SUPPORT. Necessary libraries should be linked. Please see https://github.com/google/brotli for more detail.
These folks made great contributions to polish this library to totally another level from a simple toy!
`,123)]))}const y=i(h,[["render",l]]);export{g as __pageData,y as default};
diff --git a/assets/system_httplib_README.md.WIkDaVQZ.lean.js b/assets/system_httplib_README.md.WIkDaVQZ.lean.js
new file mode 100644
index 00000000000..1fb5081d0e9
--- /dev/null
+++ b/assets/system_httplib_README.md.WIkDaVQZ.lean.js
@@ -0,0 +1,287 @@
+import{_ as i,c as a,a3 as t,o as n}from"./chunks/framework.DkhCEVKm.js";const g=JSON.parse('{"title":"cpp-httplib","description":"","frontmatter":{},"headers":[],"relativePath":"system/httplib/README.md","filePath":"system/httplib/README.md","lastUpdated":1731340314000}'),h={name:"system/httplib/README.md"};function l(k,s,p,e,E,r){return n(),a("div",null,s[0]||(s[0]=[t(`
// Mount / to ./www directory
+auto ret = svr.set_mount_point("/", "./www");
+if (!ret) {
+ // The specified base directory doesn't exist...
+}
+
+// Mount /public to ./www directory
+ret = svr.set_mount_point("/public", "./www");
+
+// Mount /public to ./www1 and ./www2 directories
+ret = svr.set_mount_point("/public", "./www1"); // 1st order to search
+ret = svr.set_mount_point("/public", "./www2"); // 2nd order to search
+
+// Remove mount /
+ret = svr.remove_mount_point("/");
+
+// Remove mount /public
+ret = svr.remove_mount_point("/public");
cpp
// User defined file extension and MIME type mappings
+svr.set_file_extension_and_mimetype_mapping("cc", "text/x-c");
+svr.set_file_extension_and_mimetype_mapping("cpp", "text/x-c");
+svr.set_file_extension_and_mimetype_mapping("hh", "text/x-h");
The followings are built-in mappings:
Extension
MIME Type
txt
text/plain
html, htm
text/html
css
text/css
jpeg, jpg
image/jpg
png
image/png
gif
image/gif
svg
image/svg+xml
ico
image/x-icon
json
application/json
pdf
application/pdf
js
application/javascript
wasm
application/wasm
xml
application/xml
xhtml
application/xhtml+xml
NOTE: These the static file server methods are not thread safe.
// Send a final status without reading the message body.
+svr.set_expect_100_continue_handler([](const Request &req, Response &res) {
+ return res.status = 401;
+});
httplib::Client cli("httpbin.org");
+
+auto res = cli.Get("/range/32", {
+ httplib::make_range_header({{1, 10}}) // 'Range: bytes=1-10'
+});
+// res->status should be 206.
+// res->body should be "bcdefghijk".
Brotli compression is available with CPPHTTPLIB_BROTLI_SUPPORT. Necessary libraries should be linked. Please see https://github.com/google/brotli for more detail.
These folks made great contributions to polish this library to totally another level from a simple toy!
`,123)]))}const y=i(h,[["render",l]]);export{g as __pageData,y as default};
diff --git a/assets/system_masking_include_readme.md.CNC1sarR.js b/assets/system_masking_include_readme.md.CNC1sarR.js
new file mode 100644
index 00000000000..11bf837832d
--- /dev/null
+++ b/assets/system_masking_include_readme.md.CNC1sarR.js
@@ -0,0 +1,93 @@
+import{_ as t,c as a,a3 as n,o as s}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Data Masking Framework","description":"","frontmatter":{},"headers":[],"relativePath":"system/masking/include/readme.md","filePath":"system/masking/include/readme.md","lastUpdated":1731340314000}'),o={name:"system/masking/include/readme.md"};function i(l,e,r,p,c,d){return s(),a("div",null,e[0]||(e[0]=[n(`
This is a high level description of the data obfuscation framework defined in system/masking/include. The framework includes a platform component, the engine, that exposes obfuscation logic supplied by dynamically loaded plugin libraries. The framework can be used to obfuscate sensitive data, such as passwords and PII. Possible use cases including preventing trace output from revealing sensitive values and partially masking values in situations where a user needs to see "enough" of a value to confirm its correctness without seeing all of the value (e.g., when a user is asked to confirm the last four digits of an account number).
File
Contents
datamasking.h
Declares the core framework interfaces. Most are used by the engine and plugins.
datamaskingengine.hpp
Defines the engine component from the core engine interface.
datamaskingplugin.hpp
Defines a set of classes derived from the core plugin interfaces that can be used to provide rudimentary obfuscation. It is expected that the some or all classes will be subclassed, if not replaced outright, in new plugins.
datamaskingshared.hpp
Defines utilities shared by engine and plugin implementations.
A domain is a representation of the obfuscation requirements applicable to a set of data.
Consider that the data used to represent individuals likely differs between countries. With different data, requirements for obfuscation may reasonably be expected to vary. Assuming that requirements do change between countries, each country's requirements could logically constitute a separate domain. The capacity to define multiple domains does not create a requirement to do so.
Requirements can change over time. To support this, a domain can be seen as a collection of requirement snapshots where each snapshot defines the complete set of requirements for the domain at a point in time. Snapshots are referenced by unique version numbers, which should be sequential starting at 1.
Obfuscation is always applied based on a single snapshot of a domain's requirements.
Domains are represented in the framework interface as text identifiers. Each distinct domain is identified by at least one unique identifier.
A masker is a provider of obfuscation for a single snapshot of a domain's requirements. There are three masking operations defined in this framework. Each instance decides which of the three it will support, and how it will support them. The three operations are:
maskValue obfuscates individual values based on snapshot-defined meanings. For example, a value identified as a password might require complete obfuscation anywhere it appears, or an account number may require complete obfuscation in some cases and partial obfuscation in others.
maskContent obfuscates a variable number of values based on context provided by surrounding text. For example, the value of an HTTP authentication header or the text between <Password> and </Password> might require obfuscation. This operation can apply to both structured and unstructured text content.
maskMarkupValue obfuscates individual values based on their locationa within an in-memory representation of a structured document, such as XML or JSON. For example, the value of element Password might require obfuscation unconditionally, while the value of element Value might require obfuscation only if a sibling element named Name exists with value password. This operation relies on the caller's ability to supply context parsed from structured content.
A masker may be either stateless or stateful. With a stateless masker, identical input will produce identical output for every requested operation. A stateful masker, however, enables its user to affect operation outputs (i.e., identical input may not produce identical output for each operation).
Maskers are represented in the framework interface using IDataMasker, with IDataMaskerInspector providing access to less frequently used information about the domain.
A profile is a stateless masker. Each instance defines the requirements of one or more snapshots of a single domain.
Snapshots are versioned. Each profile declares a minimum, maximum, and default version. Masker operations apply to the default version. Other declared versions may be accessed using a stateful context, which the profile can create on demand.
Each instance must support at least one version of a domain's requirements. Whether an instance supports more than one version depends on the implementation and on user preference. A domain can be viewed as a collection of one or more profiles where each profile defines a unique set of requirement snapshots applicable to the same underlying data.
Refer to IDataMaskingProfile (extending IDataMasker) and IDataMaskingProfileInspector (extending IDataMaskerInspector) for additional information.
A context is a stateful masker. Instantiated by and tightly coupled to a profile, it provides some user control over how masking operations are completed.
For a profile supporting multiple versions, the requirements of a non-default version may be applied.
Custom properties, defined by a profile, may be managed.
valuetype-set is a pre-defined property used to select the group of value types that may be masked by any operation.
rule-set is a pre-defined property used to select the group of rules that will be applied for maskContent requests.
Profile implementations may define additional properties as needed. For example, one might define mask-pattern to override the default obfuscation pattern to avoid replicating (and complicating) configurations just to change the appearance of obfuscated data.
Trace output produced by operation requests, including errors and warnings encountered during request processing, can be controlled per context. This does not affect the operation output, per-se, but provides compatibility with transactional trace output control.
Refer to IDataMaskingProfileContext (extending IDataMasker) and IDataMaskingProfileContextInspector (extending IDataMaskerInspector) for additional information.
Each snapshot defines one or more value types. A value type is a representation of the requirements pertaining to a particular concept of a domain datum. Requirements include:
instructions for identifying occurrences based on contextual clues found in a body of text; and
instructions for applying obfuscation to an identified value.
A Social Security Number, or SSN, is a U.S.-centric datum that requires obfuscation and for which a value type may be defined. Element names associated with an SSN may include, but are not limited to, SSN and SOCS; the value type is expected to identify all such names used within the domain. SSN occurrences are frequently partially masked, with common formats being to mask only the first four or the last five digits of the nine digit number; the value type defines which formats are available besides the default of masking all characters.
Value types are represented in the framework using IDataMaskingProfileValueType. They are accessed through the IDataMaskerInspector interface.
A mask style describes how obfuscation is applied to a value. It is always defined in the context of a value type, and a value type may define multiple.
A value type is not required to define any mask styles. If none are defined, all value characters are obfuscated. If the requested mask style is not defined, the default obfuscation occurs; the value type will not attempt to guess which of the defined styles is appropriate.
Mask styles are represented in the framework using IDataMaskingProfileMaskStyle. They are accessed through the IDataMaskingProfileValueType interface.
A rule contains the information necessary to locate at least one occurrence of a value type datum to be obfuscated. It is always defined in the context of a value type, and a value type may define multiple.
maskValue requests do not use rules. maskContent requests rely on rules for locating affected values, with the relationship between a profile and its rules an implementation detail. maskMarkupValue requests, when implemented, may also use rules or may take an entirely different approach.
Rules are represented in the framework as an abastract concept that cannot be inspected individually. Inspection may be used to establish the presence of rules, but not to examine individual instances.
The combination of a shared library and entry point function describes a plugin. The input to a plugin is a property tree describing one or more profiles. The output of an entry point function is an iterator of profiles. The profiles created by the function may all be associated with the same domain, but this is not required.
Plugin results are represented in the framework using IDataMaskingProfileIterator.
An engine is the platform's interface to obfuscation. It loads domains by loading one or more plugins. Plugins yield profiles, from which domains are inferred.
Once configured with at least one domain, a caller can obtain obfuscation in multiple ways:
As an instance of a stateless masker, an engine can provide obfuscation. The default version of the default profile of the default domain will be used, and no custom context properties can be used. This is the simplest integration, appropriate for use when the host process implicitly trusts that the default configuration is sufficient.
An engine may be used to obtain a stateless profile for a specific domain. If no version is requested, the default profile of the requested domain will be used. If a version is specified, the domainprofile supporting the requested version is used. This usage supports host processes that are aware of domains and need to use specific instances.
An engine may be used to obtain a context tied to a specific version of a domain. The default domain may be requested with an empty string. The default version may be requested by passing 0. With a context, a caller may manipulate further the environment for obfuscation requests.
Engines are represented in the framework using IDataMaskingEngine (extending IDataMasker) and IDataMaskingEngineInspector for additional information.
This section assumes a context is in use. Absent a context, only the default state of the default version of a profile can be used.
The framework reserves multiple custom property names, which are described in subsequent subsections. It also allows implementations to define additional properties using any non-reserved name.
Suppose an implementation defines a property to override the default obfuscation pattern. Let's call this property default-pattern. A caller could interrogate a masker to know if default-pattern is accepted. If accepted, setProperty("default-patterns", "#") can be used to register an override.
All custom properties, whether defined in the framework or in third party libraries, are managed using a generic context interface. This interface includes:
The abstraction includes the concept of sets of related value types. All value types should be assigned membership in at least one set. One set should be selected by default. The custom context property valuetype-set is used to select a different set.
The rationale for this is an expectation that certain data always requires obfuscation. A password, for example, would never not require obfuscation. Other data may only require obfuscacation in certain situations, such as when required by individual customer agreements. Callers should not be required to complete additional steps to act on data that must always be obfuscated, and should not need to know which data falls in which category.
The set name "*" is reserved to select all value types regardless of their defined set membership. This mechanism is intended to assist with compatibility checks, and should be used with care in other situations.
Use acceptsProperty to determine whether a snapshot recognizes a custom property name. Use usesProperty to learn if the snapshot includes references to the custom property name. Acceptance implies awareness of the set (and what it represents) even if selecting it will not change the outcome of any masking request. Usage implies that selecting the set will change the outcome of at least one masking operation.
The property valuetype-set is used to select a named set. A check of this property addresses whether the concept of a set is accepted or used by the snapshot.
For improved compatibility checks, implementations are encouraged to reserve additional property names that enable checks for individual set names. For each set name, foo, the included implementations report property valuetype-set:foo as used.
The included implementations allow membership in either a default, unnamed set or in any number of named sets. Absent a contextual request for a named set, the unnamed set is selected by default. With a contextual set request, the members of the named set are selected in addition to members of the unnamed set. Members of the unnamed set are never not selected.
Unnamed set membership cannot be defined explicitly. Because this set is always selected, assigning a value type to both a named and the unnamed set is redundant - the type will be selected whether the named set is requested or not.
Example 1a: default value type
profile:
+ valueType:
+ - name: type1
Value typetype1 belongs to the default, unnamed value type set. Property valuetype-set is accepted, but not used; an attempt to use it will change no results.
Extending the previous example, five types and four sets are in use. Property valuetype-set is both accepted and used, as it can now impact results. Properties valuetype-set:set1, valuetype-set:set2 and valuetype-set:set3 are also both accepted and used, but are intended only for use with compatibility checks.
This table shows which types are selected for each requested set:
Continuing the previous example, the profile has declared acceptance of two additional sets using properties valuetype-set:set4 and valuetype-set:set5. Selection of these sets will not change results, but a caller might accept acceptance of a set as sufficient to establish compatibility.
The abstraction includes the concept of sets of related rules. All rules should be assigned membership in at least one set. One set should be selected by default. The custom context property rule-set is used to select a different set.
The rationale for this is similar to yet different than that of value type sets. Instead of one set of rules that are always selected, backward compatibility with a proprietary legacy implementation requires that the default set be replaced by an alternate collection.
The set name "*" is reserved to select all rules regardless of their defined set membership. This does not override constraints imposed by the current value type set. This mechanism is intended to assist with compatibility checks, and should be used with care in other situations.
Use acceptsProperty to determine whether a snapshot recognizes a custom property name. Use usesProperty to learn if the snapshot includes references to the custom property name. Acceptance implies awareness of the set (and what it represents) even if selecting it will select no rules. Usage implies that selecting the set will select at least one rule.
The property rule-set is used to select a named set. A check of this property addresses whether the concept of a set is accepted or used by the snapshot.
For improved compatibility checks, implementations are encouraged to reserve additional property names that enable checks for individual set names. For each set name, foo, the included implementations report property rule-set:foo as used.
The included implementations allow membership in any number of named sets as well as a default, unnamed set. Absent a contextual request for a named set, the unnamed set is selected by default. With a contextual set request, only the members of the requested set are selected.
Unlike value type sets, the unnamed set is not always selected. Because of this difference, a rule may be assigned explicit membership in the unnamed set. This is optional for rules that belong only to the unnamed set and is required for rules meant to be selected as part of the unnamed set and one or more named sets.
The rules described here are intentionally incomplete, showing only what is necessary for this example. Four rules are defined:
The rule named foo implicitly belongs to the unnamed set.
The second rule explicitly belongs to the unnamed set.
The third rule explicitly belongs to the unnamed set and to set1.
The fourth rule belongs to set1.
property rule-set is both accepted and used. Properties rule-set: and rule-set:set1 are both accepted and used, intended for use with compatibility checks. Property rule-set:set2 is accepted.
Given a single domain datum, obfuscate the content "as needed". The framework anticipates two interpretations of "as needed", either conditional or unconditional:
Conditional means that values of a named type are only obfuscated if the named type is known by and selected in the snapshot. This usage assumes that the profile is the authoritative source for obfuscation requirements. The profile's consumer may ask for any value to be obfuscated, but the profile is not required to act.
Unconditional means that all value obfuscation requests will result in obfuscation. The named type is not required to be known by or selected in the snapshot. This usage assumes that the profile's consumer is the authoritative source for obfuscation requirements. If a consumer asks for a value to be obfuscated, the profile must act.
Each plugin will define its own interpretation of "as needed". The API distinction between conditional and unconditional is a hint intended to guide implementations capable of both, and should be ignored by implementations that are not.
The buffer, offset, and length parameters define what is assumed to be a single domain datum. That is, every character in the given character range is subject to obfuscation.
The valueType parameter determines if obfuscation is required, with conditionally controlling whether an unrecognized valueType is ignored (true) or forced to obfuscate (false).
The maskStyle parameter may be used to affect the nature of obfuscation to be applied. If the parameter names a defined mask style, that obfuscation format is applied. If the parameter does not name a define mask style, the value type's default obfuscation format is applied.
The canMaskValue method of IDataMasker can be used to establish whether a snapshot supports this operation. A result of true indicates that the plugin is capable of performing obfuscation in response to a request. Whether a combination of input parameters exists that results in obfuscation depends on the definition of the snapshot.
The hasValueType method of IDataMaskerInspector can be used to check if a given name is selected in the snapshot. The reserved name "*" may be used to detect unconditional obfuscation support when available in the snapshot. Names defined yet unselected in the snapshot are not directly detectable, but can be detected by comparing results of using multiple context configurations.
The name "*" is reserved by the abstraction for compatibility checks.
To confirm the availability of a specific mask style first requires confirmation of the value type to which the mask style is related. Use queryValueType to obtain the defined instance and, from the result, call queryMaskStyle to confirm the existence of the mask stye. For each of these, corresponding get... methods are defined to obtain new references to the objects. The methods are declared by IDataMaskerInspector.
The included implementations are inherently conditional. Obfuscation depends on matching valueType with a selected value type instance in the snapshot, where an instance is selected if it belongs to the currently selected value type set.
Unconditional obfuscation is enabled in profiles that include a value type instance named "*". If an instance with this name is selected by the currently selected value type set, all values for undefined value types will be obfuscated. Values for defined but unselected value types will not be obfuscated.
A value of type "foo" will be obfuscated when type "foo" is selected by the currently selected value type set.
A value of type "foo" will be obfuscated when type "foo" is undefined and type "*" is selected by the currently selected value type set.
A value of type "foo" will not be obfuscated when type "foo" is defined but unselected by the currently selected value type set.
This snippet declares two value types, with two total sets. The table shows which values will be conditionally obfuscated based on a the combination of valueType and the currently selected value type set.
valueType
Selected Set
Obfuscated
type1
N/A
Yes
type1
set1
Yes
type2
N/A
No
type2
set1
Yes
typeN
N/A
No
typeN
set1
No
"typeN" in the table represents any named type, excluding the reserved "*", not explicitly defined in the profile.
Extending the previous example with a third value type, values of unknown type are now obfuscated. Values of known but unselected type remain unobfuscated.
valueType
Selected Set
Obfuscated
type1
N/A
Yes
type1
set1
Yes
type2
N/A
No
type2
set1
Yes
typeN
N/A
Yes
typeN
set1
Yes
The name reserved by the abstraction to detect support for unconditional obfuscation is the same name reserved by the implementation to define that support.
"typeN" in the table represents any named type, excluding the reserved "*", not explicitly defined in the profile.
The buffer, offset, and length parameters define what is assumed to be a blob of content that may contain zero or more occurrences of domain data. The blob is expected to contain sufficient context allowing a snapshot's rules to locate any included occurrences.
The context parameter optionally influences which rules are selected in a snapshot, while the contentType parameter offers a hint as to the blob's data format. Each snapshot rule may be assigned a format to which it applies. Operation requests may be optimized by limiting the number of selected rules applied by describing the format. Selected rules not assigned a format will be applied in all requests.
There is no equivalent to the unconditional mode offered by maskValue. Content obfuscation is always conditional on matching rules.
The canMaskContent method of IDataMasker can be used to establish whether a snapshot supports this operation. A result of true indicates that the plugin is capable of performing obfuscation in response to a request. Whether a combination of input parameters exists that results in obfuscation depends on the definition of the snapshot.
The hasRule method of IDataMaskerInspector indicates if at least one rule is selected in the snapshot for a given content type. Rules not associated with any content type may be interrogated using an empty string.
It is not possible to discern any information about the selected rules, such as their associated value types or the cues used to apply them in a buffer.
The groundwork exists for two types of rules, serial and parallel. Serial rules, as the name implies, are evaluated sequentially. Parallel rules, on the other hand, are evaluated concurrently. Concurrent evaluation is expected to be more efficient than sequential, but no concurrent implementation is provided.
The included sequential implementation should be viewed as a starting point. Each rule defines a start token and an optional end token. For each occurrence of the start token in the blob, a corresponding search for an end token (newline if omitted), is performed. When both parts are found, the content between is obfuscated using the associated value type's default mask style.
Traversing a potentially large blob of text once per rule is inefficient. A concurrent implementation is in development to improve performance and capabilities. A domain originated using the original implementation and migrated to the new implementation illustrates one domain implemented by multiple plugins and, by extension, multiple profiles.
Where maskValue obfuscates individual values based on what the caller believes the value to be, and maskContent obfuscates any number of values it identifies based on cues found in surrounding text, maskMarkupValue obfuscastes individual values based on cues not provided to it.
Given a value and a relative location within a structure document, an implementation must decide whether obfuscation is required, may be required, or is not required. If required, it can obfuscate immediately. If not required, and can return immediately. If it might be required, the implementation must request the context it needs from the caller in order to make a final determination.
A request to mask the content of an element named Value might depend on the content of a sibling element named Name. If value of Value probably does not require obfuscation when the value of Name is city, but almost certainly does require obfuscation when the value of Name is password. When processing the request for Value, an implementation must ask for the value of a sibline named Name to decide whether or not to obfuscate the data.
There is no equivalent to the unconditional mode offered by maskValue. Markup value obfuscation presumes that the caller is traversing a structured document without awareness of which values require obfuscation. If the caller knows which values require obfuscation, it should use maskValue on those specific values.
The canMaskMarkupValue method of IDataMasker can be used to establish whether a snapshot supports this operation. A result of true indicates that the plugin is capable of performing obfuscation in response to a request. Whether a combination of input parameters exists that results in obfuscation depends on the definition of the snapshot.
The getMarkupValueInfo method of IDataMaskerInspector determines if the value of an element or attribute requires obfuscation. If required, the info parameter will describe on completion how to perform the obfuscation using maskValue instead of maskMarkupValue. This is done to optimize performance by eliminating the need to reevaluate rules to re-identify a match.
Assme a value type representing passwords is defined. If given an element value containing a URL with an embedded password, obfuscation is required. But most likely not for the entire value. In addition to identifying the value type associated with value, the offset and length of the embedded substring requiring obfuscation are supplied; use of maskValue with these three values and no mask style will apply the default obfuscation only to the embedded password. Alternatively, if a mask style is supplied, use of maskValue with this style and the original value offset and length should obfuscate only the embedded password.
A callback interface is required for all calls to getMarkupValueInfo. The interface is only used when additional document context is required to make a determination about the requested value. In these cases, the checks are proximity-dependent and cannot be performed as part of load-time compatibility checks, when no document is available.
Use of runtime compatibility checks may be unavoidable in some circumstances, but reliance on this in every script is inefficient. It may also devalue trace output by omitting messaging required to debug an issue, a problem that might only be found when said messaging is needed.
The requirements parameter represents a standardized set of runtime compatibility checks to be applied. By default, each check must pass to establish compatibility. Explicitly optional checks may be included to detect conditions the caller is prepared to work around.
This can test which values will or won't be affected by maskValue, and which obfuscation styles are available. It can test which rule sets will impact maskContent. It specifically excludes maskMarkupValue checks that depend on the proximity of one value to another.
Returning to the earlier SSN example, a caller might require that an SSN value type will be obfuscated. It might also want to use a particilar mask style, but may be prepared for its absence. A caller may prefer to mask the first four digits but may accept masking the last five instead, or may omit certain trace messages.
compatibility:
+ - context:
+ - domain: optional text
+ version: optional number
+ property:
+ - name: required text
+ value: required text
+ accepts:
+ - name: required text
+ presence: one of "r", "o", or "p"
+ uses:
+ - name: required text
+ presence: one of "r", "o", or "p"
+ operation:
+ - name: one of "maskValue", "maskContent", or "maskMarkupValue"
+ presence: one of "r", "o", or "p"
+ valueType
+ - name: required text
+ presence: one of "r", "o", or "p"
+ maskStyle:
+ - name: required text
+ presence: one of "r", "o", or "p"
+ Set:
+ - name: required text
+ presence: one of "r", "o", or "p"
+ rule:
+ - contentType: required text
+ presence: one of "r", "o", or "p"
The requirements parameter of checkCompatibility must be either a compatibility element or the parent of a collection of compatibility elements. Each instance may contain at most one context element, three operation elements (one per operation), and a variable number of accepts, uses, valueType, or rule elements.
The context element describes the target of an evaluation. It may include a domain identifier, or assume the default domain. It may include a version number, or assume the default snapshot of the domain. It may specify any number of custom context properties to select various profile elements in the snapshot.
The accepts and uses elements identify custom context property names either accepted or used within a snapshot. Acceptance indicates some part of a snapshot has declared an understanding of the named property. Usage indicates the some part of a snapshot will react to the named value being set. Usage implies acceptance.
To improve compatibility check capabilities, a snapshot may synthesize properties that, if set, will have no effect. Specifically, a caller may be more interested to know if a particular set name (for either value type or rule set membership) is used than to know that an unidentified set name is used.
The Operations element identifies which operations must be supported. Its purpose is to enforce availability of an operation when no more explicit requirements are given (for example, one may require maskContent without also requiring the presence of rules for any given content type). When more explicit requirements are given, the attributes of this element are redundant. If all attributes are redundant, the element may be omitted.
The valueType element describes all optional and mandatory requirements for using maskValue with a single value type name and optional mask style name. Set membership requirements can also be evaluated, which is most useful when compatibility/context/property/@name is "*".
The rule element describes all optional and mandatory requirements for using maskContent with a single content type value. Unlike evaluable value type set membership, rule set membership requirements cannot be evaluated.
In all elements described as accepting @presence, the acceptable values are
r: the check is required to pass
o: the check is optional
p: the check is prohibited from passing
`,170)]))}const h=t(o,[["render",i]]);export{u as __pageData,h as default};
diff --git a/assets/system_masking_include_readme.md.CNC1sarR.lean.js b/assets/system_masking_include_readme.md.CNC1sarR.lean.js
new file mode 100644
index 00000000000..11bf837832d
--- /dev/null
+++ b/assets/system_masking_include_readme.md.CNC1sarR.lean.js
@@ -0,0 +1,93 @@
+import{_ as t,c as a,a3 as n,o as s}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"Data Masking Framework","description":"","frontmatter":{},"headers":[],"relativePath":"system/masking/include/readme.md","filePath":"system/masking/include/readme.md","lastUpdated":1731340314000}'),o={name:"system/masking/include/readme.md"};function i(l,e,r,p,c,d){return s(),a("div",null,e[0]||(e[0]=[n(`
This is a high level description of the data obfuscation framework defined in system/masking/include. The framework includes a platform component, the engine, that exposes obfuscation logic supplied by dynamically loaded plugin libraries. The framework can be used to obfuscate sensitive data, such as passwords and PII. Possible use cases including preventing trace output from revealing sensitive values and partially masking values in situations where a user needs to see "enough" of a value to confirm its correctness without seeing all of the value (e.g., when a user is asked to confirm the last four digits of an account number).
File
Contents
datamasking.h
Declares the core framework interfaces. Most are used by the engine and plugins.
datamaskingengine.hpp
Defines the engine component from the core engine interface.
datamaskingplugin.hpp
Defines a set of classes derived from the core plugin interfaces that can be used to provide rudimentary obfuscation. It is expected that the some or all classes will be subclassed, if not replaced outright, in new plugins.
datamaskingshared.hpp
Defines utilities shared by engine and plugin implementations.
A domain is a representation of the obfuscation requirements applicable to a set of data.
Consider that the data used to represent individuals likely differs between countries. With different data, requirements for obfuscation may reasonably be expected to vary. Assuming that requirements do change between countries, each country's requirements could logically constitute a separate domain. The capacity to define multiple domains does not create a requirement to do so.
Requirements can change over time. To support this, a domain can be seen as a collection of requirement snapshots where each snapshot defines the complete set of requirements for the domain at a point in time. Snapshots are referenced by unique version numbers, which should be sequential starting at 1.
Obfuscation is always applied based on a single snapshot of a domain's requirements.
Domains are represented in the framework interface as text identifiers. Each distinct domain is identified by at least one unique identifier.
A masker is a provider of obfuscation for a single snapshot of a domain's requirements. There are three masking operations defined in this framework. Each instance decides which of the three it will support, and how it will support them. The three operations are:
maskValue obfuscates individual values based on snapshot-defined meanings. For example, a value identified as a password might require complete obfuscation anywhere it appears, or an account number may require complete obfuscation in some cases and partial obfuscation in others.
maskContent obfuscates a variable number of values based on context provided by surrounding text. For example, the value of an HTTP authentication header or the text between <Password> and </Password> might require obfuscation. This operation can apply to both structured and unstructured text content.
maskMarkupValue obfuscates individual values based on their locationa within an in-memory representation of a structured document, such as XML or JSON. For example, the value of element Password might require obfuscation unconditionally, while the value of element Value might require obfuscation only if a sibling element named Name exists with value password. This operation relies on the caller's ability to supply context parsed from structured content.
A masker may be either stateless or stateful. With a stateless masker, identical input will produce identical output for every requested operation. A stateful masker, however, enables its user to affect operation outputs (i.e., identical input may not produce identical output for each operation).
Maskers are represented in the framework interface using IDataMasker, with IDataMaskerInspector providing access to less frequently used information about the domain.
A profile is a stateless masker. Each instance defines the requirements of one or more snapshots of a single domain.
Snapshots are versioned. Each profile declares a minimum, maximum, and default version. Masker operations apply to the default version. Other declared versions may be accessed using a stateful context, which the profile can create on demand.
Each instance must support at least one version of a domain's requirements. Whether an instance supports more than one version depends on the implementation and on user preference. A domain can be viewed as a collection of one or more profiles where each profile defines a unique set of requirement snapshots applicable to the same underlying data.
Refer to IDataMaskingProfile (extending IDataMasker) and IDataMaskingProfileInspector (extending IDataMaskerInspector) for additional information.
A context is a stateful masker. Instantiated by and tightly coupled to a profile, it provides some user control over how masking operations are completed.
For a profile supporting multiple versions, the requirements of a non-default version may be applied.
Custom properties, defined by a profile, may be managed.
valuetype-set is a pre-defined property used to select the group of value types that may be masked by any operation.
rule-set is a pre-defined property used to select the group of rules that will be applied for maskContent requests.
Profile implementations may define additional properties as needed. For example, one might define mask-pattern to override the default obfuscation pattern to avoid replicating (and complicating) configurations just to change the appearance of obfuscated data.
Trace output produced by operation requests, including errors and warnings encountered during request processing, can be controlled per context. This does not affect the operation output, per-se, but provides compatibility with transactional trace output control.
Refer to IDataMaskingProfileContext (extending IDataMasker) and IDataMaskingProfileContextInspector (extending IDataMaskerInspector) for additional information.
Each snapshot defines one or more value types. A value type is a representation of the requirements pertaining to a particular concept of a domain datum. Requirements include:
instructions for identifying occurrences based on contextual clues found in a body of text; and
instructions for applying obfuscation to an identified value.
A Social Security Number, or SSN, is a U.S.-centric datum that requires obfuscation and for which a value type may be defined. Element names associated with an SSN may include, but are not limited to, SSN and SOCS; the value type is expected to identify all such names used within the domain. SSN occurrences are frequently partially masked, with common formats being to mask only the first four or the last five digits of the nine digit number; the value type defines which formats are available besides the default of masking all characters.
Value types are represented in the framework using IDataMaskingProfileValueType. They are accessed through the IDataMaskerInspector interface.
A mask style describes how obfuscation is applied to a value. It is always defined in the context of a value type, and a value type may define multiple.
A value type is not required to define any mask styles. If none are defined, all value characters are obfuscated. If the requested mask style is not defined, the default obfuscation occurs; the value type will not attempt to guess which of the defined styles is appropriate.
Mask styles are represented in the framework using IDataMaskingProfileMaskStyle. They are accessed through the IDataMaskingProfileValueType interface.
A rule contains the information necessary to locate at least one occurrence of a value type datum to be obfuscated. It is always defined in the context of a value type, and a value type may define multiple.
maskValue requests do not use rules. maskContent requests rely on rules for locating affected values, with the relationship between a profile and its rules an implementation detail. maskMarkupValue requests, when implemented, may also use rules or may take an entirely different approach.
Rules are represented in the framework as an abastract concept that cannot be inspected individually. Inspection may be used to establish the presence of rules, but not to examine individual instances.
The combination of a shared library and entry point function describes a plugin. The input to a plugin is a property tree describing one or more profiles. The output of an entry point function is an iterator of profiles. The profiles created by the function may all be associated with the same domain, but this is not required.
Plugin results are represented in the framework using IDataMaskingProfileIterator.
An engine is the platform's interface to obfuscation. It loads domains by loading one or more plugins. Plugins yield profiles, from which domains are inferred.
Once configured with at least one domain, a caller can obtain obfuscation in multiple ways:
As an instance of a stateless masker, an engine can provide obfuscation. The default version of the default profile of the default domain will be used, and no custom context properties can be used. This is the simplest integration, appropriate for use when the host process implicitly trusts that the default configuration is sufficient.
An engine may be used to obtain a stateless profile for a specific domain. If no version is requested, the default profile of the requested domain will be used. If a version is specified, the domainprofile supporting the requested version is used. This usage supports host processes that are aware of domains and need to use specific instances.
An engine may be used to obtain a context tied to a specific version of a domain. The default domain may be requested with an empty string. The default version may be requested by passing 0. With a context, a caller may manipulate further the environment for obfuscation requests.
Engines are represented in the framework using IDataMaskingEngine (extending IDataMasker) and IDataMaskingEngineInspector for additional information.
This section assumes a context is in use. Absent a context, only the default state of the default version of a profile can be used.
The framework reserves multiple custom property names, which are described in subsequent subsections. It also allows implementations to define additional properties using any non-reserved name.
Suppose an implementation defines a property to override the default obfuscation pattern. Let's call this property default-pattern. A caller could interrogate a masker to know if default-pattern is accepted. If accepted, setProperty("default-patterns", "#") can be used to register an override.
All custom properties, whether defined in the framework or in third party libraries, are managed using a generic context interface. This interface includes:
The abstraction includes the concept of sets of related value types. All value types should be assigned membership in at least one set. One set should be selected by default. The custom context property valuetype-set is used to select a different set.
The rationale for this is an expectation that certain data always requires obfuscation. A password, for example, would never not require obfuscation. Other data may only require obfuscacation in certain situations, such as when required by individual customer agreements. Callers should not be required to complete additional steps to act on data that must always be obfuscated, and should not need to know which data falls in which category.
The set name "*" is reserved to select all value types regardless of their defined set membership. This mechanism is intended to assist with compatibility checks, and should be used with care in other situations.
Use acceptsProperty to determine whether a snapshot recognizes a custom property name. Use usesProperty to learn if the snapshot includes references to the custom property name. Acceptance implies awareness of the set (and what it represents) even if selecting it will not change the outcome of any masking request. Usage implies that selecting the set will change the outcome of at least one masking operation.
The property valuetype-set is used to select a named set. A check of this property addresses whether the concept of a set is accepted or used by the snapshot.
For improved compatibility checks, implementations are encouraged to reserve additional property names that enable checks for individual set names. For each set name, foo, the included implementations report property valuetype-set:foo as used.
The included implementations allow membership in either a default, unnamed set or in any number of named sets. Absent a contextual request for a named set, the unnamed set is selected by default. With a contextual set request, the members of the named set are selected in addition to members of the unnamed set. Members of the unnamed set are never not selected.
Unnamed set membership cannot be defined explicitly. Because this set is always selected, assigning a value type to both a named and the unnamed set is redundant - the type will be selected whether the named set is requested or not.
Example 1a: default value type
profile:
+ valueType:
+ - name: type1
Value typetype1 belongs to the default, unnamed value type set. Property valuetype-set is accepted, but not used; an attempt to use it will change no results.
Extending the previous example, five types and four sets are in use. Property valuetype-set is both accepted and used, as it can now impact results. Properties valuetype-set:set1, valuetype-set:set2 and valuetype-set:set3 are also both accepted and used, but are intended only for use with compatibility checks.
This table shows which types are selected for each requested set:
Continuing the previous example, the profile has declared acceptance of two additional sets using properties valuetype-set:set4 and valuetype-set:set5. Selection of these sets will not change results, but a caller might accept acceptance of a set as sufficient to establish compatibility.
The abstraction includes the concept of sets of related rules. All rules should be assigned membership in at least one set. One set should be selected by default. The custom context property rule-set is used to select a different set.
The rationale for this is similar to yet different than that of value type sets. Instead of one set of rules that are always selected, backward compatibility with a proprietary legacy implementation requires that the default set be replaced by an alternate collection.
The set name "*" is reserved to select all rules regardless of their defined set membership. This does not override constraints imposed by the current value type set. This mechanism is intended to assist with compatibility checks, and should be used with care in other situations.
Use acceptsProperty to determine whether a snapshot recognizes a custom property name. Use usesProperty to learn if the snapshot includes references to the custom property name. Acceptance implies awareness of the set (and what it represents) even if selecting it will select no rules. Usage implies that selecting the set will select at least one rule.
The property rule-set is used to select a named set. A check of this property addresses whether the concept of a set is accepted or used by the snapshot.
For improved compatibility checks, implementations are encouraged to reserve additional property names that enable checks for individual set names. For each set name, foo, the included implementations report property rule-set:foo as used.
The included implementations allow membership in any number of named sets as well as a default, unnamed set. Absent a contextual request for a named set, the unnamed set is selected by default. With a contextual set request, only the members of the requested set are selected.
Unlike value type sets, the unnamed set is not always selected. Because of this difference, a rule may be assigned explicit membership in the unnamed set. This is optional for rules that belong only to the unnamed set and is required for rules meant to be selected as part of the unnamed set and one or more named sets.
The rules described here are intentionally incomplete, showing only what is necessary for this example. Four rules are defined:
The rule named foo implicitly belongs to the unnamed set.
The second rule explicitly belongs to the unnamed set.
The third rule explicitly belongs to the unnamed set and to set1.
The fourth rule belongs to set1.
property rule-set is both accepted and used. Properties rule-set: and rule-set:set1 are both accepted and used, intended for use with compatibility checks. Property rule-set:set2 is accepted.
Given a single domain datum, obfuscate the content "as needed". The framework anticipates two interpretations of "as needed", either conditional or unconditional:
Conditional means that values of a named type are only obfuscated if the named type is known by and selected in the snapshot. This usage assumes that the profile is the authoritative source for obfuscation requirements. The profile's consumer may ask for any value to be obfuscated, but the profile is not required to act.
Unconditional means that all value obfuscation requests will result in obfuscation. The named type is not required to be known by or selected in the snapshot. This usage assumes that the profile's consumer is the authoritative source for obfuscation requirements. If a consumer asks for a value to be obfuscated, the profile must act.
Each plugin will define its own interpretation of "as needed". The API distinction between conditional and unconditional is a hint intended to guide implementations capable of both, and should be ignored by implementations that are not.
The buffer, offset, and length parameters define what is assumed to be a single domain datum. That is, every character in the given character range is subject to obfuscation.
The valueType parameter determines if obfuscation is required, with conditionally controlling whether an unrecognized valueType is ignored (true) or forced to obfuscate (false).
The maskStyle parameter may be used to affect the nature of obfuscation to be applied. If the parameter names a defined mask style, that obfuscation format is applied. If the parameter does not name a define mask style, the value type's default obfuscation format is applied.
The canMaskValue method of IDataMasker can be used to establish whether a snapshot supports this operation. A result of true indicates that the plugin is capable of performing obfuscation in response to a request. Whether a combination of input parameters exists that results in obfuscation depends on the definition of the snapshot.
The hasValueType method of IDataMaskerInspector can be used to check if a given name is selected in the snapshot. The reserved name "*" may be used to detect unconditional obfuscation support when available in the snapshot. Names defined yet unselected in the snapshot are not directly detectable, but can be detected by comparing results of using multiple context configurations.
The name "*" is reserved by the abstraction for compatibility checks.
To confirm the availability of a specific mask style first requires confirmation of the value type to which the mask style is related. Use queryValueType to obtain the defined instance and, from the result, call queryMaskStyle to confirm the existence of the mask stye. For each of these, corresponding get... methods are defined to obtain new references to the objects. The methods are declared by IDataMaskerInspector.
The included implementations are inherently conditional. Obfuscation depends on matching valueType with a selected value type instance in the snapshot, where an instance is selected if it belongs to the currently selected value type set.
Unconditional obfuscation is enabled in profiles that include a value type instance named "*". If an instance with this name is selected by the currently selected value type set, all values for undefined value types will be obfuscated. Values for defined but unselected value types will not be obfuscated.
A value of type "foo" will be obfuscated when type "foo" is selected by the currently selected value type set.
A value of type "foo" will be obfuscated when type "foo" is undefined and type "*" is selected by the currently selected value type set.
A value of type "foo" will not be obfuscated when type "foo" is defined but unselected by the currently selected value type set.
This snippet declares two value types, with two total sets. The table shows which values will be conditionally obfuscated based on a the combination of valueType and the currently selected value type set.
valueType
Selected Set
Obfuscated
type1
N/A
Yes
type1
set1
Yes
type2
N/A
No
type2
set1
Yes
typeN
N/A
No
typeN
set1
No
"typeN" in the table represents any named type, excluding the reserved "*", not explicitly defined in the profile.
Extending the previous example with a third value type, values of unknown type are now obfuscated. Values of known but unselected type remain unobfuscated.
valueType
Selected Set
Obfuscated
type1
N/A
Yes
type1
set1
Yes
type2
N/A
No
type2
set1
Yes
typeN
N/A
Yes
typeN
set1
Yes
The name reserved by the abstraction to detect support for unconditional obfuscation is the same name reserved by the implementation to define that support.
"typeN" in the table represents any named type, excluding the reserved "*", not explicitly defined in the profile.
The buffer, offset, and length parameters define what is assumed to be a blob of content that may contain zero or more occurrences of domain data. The blob is expected to contain sufficient context allowing a snapshot's rules to locate any included occurrences.
The context parameter optionally influences which rules are selected in a snapshot, while the contentType parameter offers a hint as to the blob's data format. Each snapshot rule may be assigned a format to which it applies. Operation requests may be optimized by limiting the number of selected rules applied by describing the format. Selected rules not assigned a format will be applied in all requests.
There is no equivalent to the unconditional mode offered by maskValue. Content obfuscation is always conditional on matching rules.
The canMaskContent method of IDataMasker can be used to establish whether a snapshot supports this operation. A result of true indicates that the plugin is capable of performing obfuscation in response to a request. Whether a combination of input parameters exists that results in obfuscation depends on the definition of the snapshot.
The hasRule method of IDataMaskerInspector indicates if at least one rule is selected in the snapshot for a given content type. Rules not associated with any content type may be interrogated using an empty string.
It is not possible to discern any information about the selected rules, such as their associated value types or the cues used to apply them in a buffer.
The groundwork exists for two types of rules, serial and parallel. Serial rules, as the name implies, are evaluated sequentially. Parallel rules, on the other hand, are evaluated concurrently. Concurrent evaluation is expected to be more efficient than sequential, but no concurrent implementation is provided.
The included sequential implementation should be viewed as a starting point. Each rule defines a start token and an optional end token. For each occurrence of the start token in the blob, a corresponding search for an end token (newline if omitted), is performed. When both parts are found, the content between is obfuscated using the associated value type's default mask style.
Traversing a potentially large blob of text once per rule is inefficient. A concurrent implementation is in development to improve performance and capabilities. A domain originated using the original implementation and migrated to the new implementation illustrates one domain implemented by multiple plugins and, by extension, multiple profiles.
Where maskValue obfuscates individual values based on what the caller believes the value to be, and maskContent obfuscates any number of values it identifies based on cues found in surrounding text, maskMarkupValue obfuscastes individual values based on cues not provided to it.
Given a value and a relative location within a structure document, an implementation must decide whether obfuscation is required, may be required, or is not required. If required, it can obfuscate immediately. If not required, and can return immediately. If it might be required, the implementation must request the context it needs from the caller in order to make a final determination.
A request to mask the content of an element named Value might depend on the content of a sibling element named Name. If value of Value probably does not require obfuscation when the value of Name is city, but almost certainly does require obfuscation when the value of Name is password. When processing the request for Value, an implementation must ask for the value of a sibline named Name to decide whether or not to obfuscate the data.
There is no equivalent to the unconditional mode offered by maskValue. Markup value obfuscation presumes that the caller is traversing a structured document without awareness of which values require obfuscation. If the caller knows which values require obfuscation, it should use maskValue on those specific values.
The canMaskMarkupValue method of IDataMasker can be used to establish whether a snapshot supports this operation. A result of true indicates that the plugin is capable of performing obfuscation in response to a request. Whether a combination of input parameters exists that results in obfuscation depends on the definition of the snapshot.
The getMarkupValueInfo method of IDataMaskerInspector determines if the value of an element or attribute requires obfuscation. If required, the info parameter will describe on completion how to perform the obfuscation using maskValue instead of maskMarkupValue. This is done to optimize performance by eliminating the need to reevaluate rules to re-identify a match.
Assme a value type representing passwords is defined. If given an element value containing a URL with an embedded password, obfuscation is required. But most likely not for the entire value. In addition to identifying the value type associated with value, the offset and length of the embedded substring requiring obfuscation are supplied; use of maskValue with these three values and no mask style will apply the default obfuscation only to the embedded password. Alternatively, if a mask style is supplied, use of maskValue with this style and the original value offset and length should obfuscate only the embedded password.
A callback interface is required for all calls to getMarkupValueInfo. The interface is only used when additional document context is required to make a determination about the requested value. In these cases, the checks are proximity-dependent and cannot be performed as part of load-time compatibility checks, when no document is available.
Use of runtime compatibility checks may be unavoidable in some circumstances, but reliance on this in every script is inefficient. It may also devalue trace output by omitting messaging required to debug an issue, a problem that might only be found when said messaging is needed.
The requirements parameter represents a standardized set of runtime compatibility checks to be applied. By default, each check must pass to establish compatibility. Explicitly optional checks may be included to detect conditions the caller is prepared to work around.
This can test which values will or won't be affected by maskValue, and which obfuscation styles are available. It can test which rule sets will impact maskContent. It specifically excludes maskMarkupValue checks that depend on the proximity of one value to another.
Returning to the earlier SSN example, a caller might require that an SSN value type will be obfuscated. It might also want to use a particilar mask style, but may be prepared for its absence. A caller may prefer to mask the first four digits but may accept masking the last five instead, or may omit certain trace messages.
compatibility:
+ - context:
+ - domain: optional text
+ version: optional number
+ property:
+ - name: required text
+ value: required text
+ accepts:
+ - name: required text
+ presence: one of "r", "o", or "p"
+ uses:
+ - name: required text
+ presence: one of "r", "o", or "p"
+ operation:
+ - name: one of "maskValue", "maskContent", or "maskMarkupValue"
+ presence: one of "r", "o", or "p"
+ valueType
+ - name: required text
+ presence: one of "r", "o", or "p"
+ maskStyle:
+ - name: required text
+ presence: one of "r", "o", or "p"
+ Set:
+ - name: required text
+ presence: one of "r", "o", or "p"
+ rule:
+ - contentType: required text
+ presence: one of "r", "o", or "p"
The requirements parameter of checkCompatibility must be either a compatibility element or the parent of a collection of compatibility elements. Each instance may contain at most one context element, three operation elements (one per operation), and a variable number of accepts, uses, valueType, or rule elements.
The context element describes the target of an evaluation. It may include a domain identifier, or assume the default domain. It may include a version number, or assume the default snapshot of the domain. It may specify any number of custom context properties to select various profile elements in the snapshot.
The accepts and uses elements identify custom context property names either accepted or used within a snapshot. Acceptance indicates some part of a snapshot has declared an understanding of the named property. Usage indicates the some part of a snapshot will react to the named value being set. Usage implies acceptance.
To improve compatibility check capabilities, a snapshot may synthesize properties that, if set, will have no effect. Specifically, a caller may be more interested to know if a particular set name (for either value type or rule set membership) is used than to know that an unidentified set name is used.
The Operations element identifies which operations must be supported. Its purpose is to enforce availability of an operation when no more explicit requirements are given (for example, one may require maskContent without also requiring the presence of rules for any given content type). When more explicit requirements are given, the attributes of this element are redundant. If all attributes are redundant, the element may be omitted.
The valueType element describes all optional and mandatory requirements for using maskValue with a single value type name and optional mask style name. Set membership requirements can also be evaluated, which is most useful when compatibility/context/property/@name is "*".
The rule element describes all optional and mandatory requirements for using maskContent with a single content type value. Unlike evaluable value type set membership, rule set membership requirements cannot be evaluated.
In all elements described as accepting @presence, the acceptable values are
r: the check is required to pass
o: the check is optional
p: the check is prohibited from passing
`,170)]))}const h=t(o,[["render",i]]);export{u as __pageData,h as default};
diff --git a/assets/system_masking_plugins_datamasker_readme.md.DKeIe1BA.js b/assets/system_masking_plugins_datamasker_readme.md.DKeIe1BA.js
new file mode 100644
index 00000000000..5a7408709c3
--- /dev/null
+++ b/assets/system_masking_plugins_datamasker_readme.md.DKeIe1BA.js
@@ -0,0 +1,24 @@
+import{_ as t,c as i,a3 as n,o as a}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"","description":"","frontmatter":{},"headers":[],"relativePath":"system/masking/plugins/datamasker/readme.md","filePath":"system/masking/plugins/datamasker/readme.md","lastUpdated":1731340314000}'),o={name:"system/masking/plugins/datamasker/readme.md"};function s(r,e,l,m,d,p){return a(),i("div",null,e[0]||(e[0]=[n(`
This implements a plugin library producing instances of IDataMaskingProfileIterator to be used by an IDataMaskingEngine instance. The library is written in C++, and this section is organized according to the namespace and class names used.
Standard implementation of IDataMaskingProfileContext and IDataMaskingProfileContextInspector tightly coupled to a specific version of a specific profile instance. It manages custom properties that are accepted by the associated profile, and rejects attempts to set those not accepted. The rejection is one way of letting the caller know that something it might deem essential is not handled by the profile.
Depends on CProfile, an abstract implementation of IDataMaskingProfile and IDataMaskingProfileInspector.
Implementation of IDataMaskingProfileMaskStyle providing customized masked output of values. Configuration options include:
@maximumVersion is an optional version number, needed only when the style does not apply to the maximum version of the value type in which it is defined.
@minimumVersion is an optional version number, needed only when the style does not apply to the minumum version of the value type in which it is defined.
@name is a required unique identifier for the style.
@overrideDefault is an optional Boolean flag controlling whether the style should replace the default style for the value type in which it is defined.
@pattern is an optional character sequence (one or more characters in length) that will be used to mask value content.
Am extension of CMaskStyle, this is intended to partially mask values, such as account numbers or telephone numbers, without assuming knowledge of the values. Configuration is generally expressed as:
@action what to do, either keep or mask (default); and
@location where to do it, either first or last (default); and
@count how many characters to examine, a positive integer value; and
@characters type of characters to be considered, either numbers, letters, alphanumeric, or all (default)
All four values may be omitted when only the inherited options are needed. If any of the four values are given, count is required and the other three are optional.
A value substring containing at most @count instances of the character class denoted by @characters is identified. Character classes are ASCII numeric characters (numbers), ASCII alphabetic characters (letters), ASCII alphabetic and numeric characters (alphanumeric), or any characters (all).
The value substring is at the start of the value when @location is first, and at the end of the value when @location is last.
The value substring is masked when @action is mask. All of the value with the exception of the substring is masked when @action is keep.
A base class for rule implementations, this defines standard rule properties without providing any content knowledge for use with maskContent.
Configuration includes:
@contentType is an optional, user-defined, label describing the content format to which the rule should be applied. maskContent requests that indicate a content type should apply all rules that match the type in addition to all rules that omit a type.
memberOf is an optional and repeating element identifying user-defined set labels. Only rules with membership in the requested set are considered; in the absence of an explictly requested set, a default set with an empty name is implied.
memberOf/@name is a required user-defined set label.
rule instances will frequently be specific to a content format. For example, a rule applied to XML markup value may include dependencies on XML syntax which are not applicable when masking JSON content. Rules specific to a markup format can be associated with that format using @contentType. Requests to mask content of a type will apply all rules without a @contentType value or with a matching value, and will exclude all rules with a non-matching value.
An extension of CRule, this identifies content substrings to be masked based on matching start and end tokens in the content buffer. For each occurrence of a configured start token that is balanced by a corresponding configured end token, the characters between the tokens are masked.
@endToken is an optional character sequence expected to immediately follow a content substring to be masked. This might be an XML element end tag, or a terminating double quote for a JSON value. A newline, i.e., \\n, is assumed if omitted or empty, for masking values such as HTTP headers.
@matchCase is an optional Boolean flag controlling whether token matches are case sensitive (true) or insensitivie (false). Matches are case insensitive by default. The implementation is based on ASCII data; case sensitive comparisons of similarly encoded non-ASCI text can work, but case insensitive comparisons are not supported.
@startToken is a required character sequence expected to precede a content substring to be masked.
No content type knowledge is implied by this class. An instance with @contentType of xml does not inherently know how to find values in XML markup. The defined tokens must include characters such as < and > to match an element name and quotes to match attribute values.
An implementation of IDataMaskingProfileIterator used for transforming a configuration property tree into a collection of profiles to be returned by a library entry point function. Created profiles are of the same class, identified by the template parameter profile_t.
Configuration includes:
profile An optional and repeating element defining a profile. The content of this element depends on profile_t.
If and only if profile is not included as a child of the configuration node, the node itself will be treated as a configuration for profile_t. The name of the configuration node is not required to be profile.
A concrete extension of CProfile, this manages value types and rules, providing a default implementation of maskValue but leaving other operations to subclasses.
Template parameters are:
valuetype_t identifies the concrete implementation of IDataMaskingProfileValueType to be instantiated during configuration.
rule_t identifies the concrete implementation of rules to be managed. Creation is assumed to be the responsibility of the value type.
context_t identifies the concrete implementation of IDataMaskingProfileContext to be instantiated on demand.
Configuration includes:
@defaultVersion is an optional version number identifying the version to be used when version 0 is requested. Omission or 0 implies the maximum version (which is configured before this value).
@domain is the required default identifier used to select this profile.
legacyDomain is an optional and repeating element identifying alternate identifiers by which the profile may be selected. This enables domain identifier naming conventions to be changed without breaking pre-existing references.
legacyDomain/@id is a required identifier of this profile.
@maximumVersion is an optional version number identifying the highest version supported by the configuration. Omission or 0 implies a value matching the minimum version (whether implicitly or explicitly defined).
@minimumVersion is an optional version number identifying the lowest version supported by the configuration. Omission or 0 implies 1.
@name is an optional label for the instance. If given, the value is used in trace output.
property is an optional and repeating element describing a custom context property recognized by the profile. The profile can declare awareness of properties without explicit use of them.
property/@name is a required context property name.
property/@minimumVersion is an optional version number indicating the lowest profile version aware of the property. Omission or 0 implies the profile minimum.
property/@maximumVersion is an optional version number indicating the highest profile version aware of the property. Omisssion or 0 implies the profile maximum.
valueType is an optional and repeating element describing the value types defined by the profile. Element content depends on the value of valuetype_t.
For profiles supporting a single version, value type names must be unique. For profiles supporting multiple versions, value type names may be repeated but must be unique for each version. To illustrate, consider this snippet:
In the example, foo and bar are each defined twice. The redefinition of foo is acceptable because each instance applies to a different version. The redefinition of bar is invalid because both instances claim to apply to version 2.
The value type name * is reserved by this class. A profile that includes a value type named * supports unconditional value masking. maskValue requests specifying unknown value type names can fall back to this special type and force masking for these values. Without this type, maskValue masks what it thinks should be masked; with this type, it trusts the caller to request masking for only those values known to require it. Value types included in unselected value type sets are known to the profile and, as such, can be used to prevent specific typed values from ever being masked.
The requirement of a type definition instead of a simpler flag is to enable the definition of mask styles. It has the side effect of enabling the definition of rules. In theory, an entire profile could be defined using a single value type. This may make sense in some cases, but not in all. For example, a partial mask style intended for use with U.S. Social Security numbers could be inappropriately applied to a password. Use care when configuring this type.
An extension of TProfile that adds support for maskContent. It is assumed by maskContent that each applicable rule must be applied serially, i.e., one after the other, using an bool applyRule(buffer, length, context) interface.
Template parameters are unchanged from TProfile.
Configuration options are unchanged from TProfile.
An implementation of IDataMaskingProfileValueType that manages mask styles and creates rules for the profile.
Template parameters are:
maskstyle_t identifies the concrete implementation of IDataMaskingProfileMaskStyle to be instantiated during configuration.
rule_t identifies the concrete implementation of rules to be created during configuration.
Configuration includes:
@maximumVersion is an optional version number, needed only when the type does not apply to the maximum version of the profile in which it is defined.
@minimumVersion is an optional version number, needed only when the type does not apply to the minumum version of the profile in which it is defined.
maskStyle is an optional and repeating element describing mask styles defined by the type. Element content depends on the value of maskstyle_t.
memberOf is an optional and repeating element identifying user-defined set labels. All value types that are not explicitly assigned set membership are considered for all masking requests. Value types assigned set membership are considered only when one of their assigned sets is selected with the request context.
memberOf/@name is a required user-defined set label.
@name is a required unique identifier for the type.
rule is an optional and repeating element describing rules defined by the type. Element content depends on the value of rule_t.
For value types supporting a single version, mask style names must be unique. For types supporting multiple versions, style names may be repeated but must be unique for each version. To illustrate, consider this snippet:
In the example, foo and bar are each defined twice. The redefinition of foo is acceptable because each instance applies to a different version. The redefinition of bar is invalid because both instances claim to apply to version 2.
This section describes the entry point functions exported by the shared library. The library must export one function, and may export multiple functions.
The description of each entry point will identify which of the previously described classes is used to represent the returned collection of profiles. If a templated class is identified, the template parameters are also listed. Refer to the class descriptions for additional information.
`,63)]))}const h=t(o,[["render",s]]);export{u as __pageData,h as default};
diff --git a/assets/system_masking_plugins_datamasker_readme.md.DKeIe1BA.lean.js b/assets/system_masking_plugins_datamasker_readme.md.DKeIe1BA.lean.js
new file mode 100644
index 00000000000..5a7408709c3
--- /dev/null
+++ b/assets/system_masking_plugins_datamasker_readme.md.DKeIe1BA.lean.js
@@ -0,0 +1,24 @@
+import{_ as t,c as i,a3 as n,o as a}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"","description":"","frontmatter":{},"headers":[],"relativePath":"system/masking/plugins/datamasker/readme.md","filePath":"system/masking/plugins/datamasker/readme.md","lastUpdated":1731340314000}'),o={name:"system/masking/plugins/datamasker/readme.md"};function s(r,e,l,m,d,p){return a(),i("div",null,e[0]||(e[0]=[n(`
This implements a plugin library producing instances of IDataMaskingProfileIterator to be used by an IDataMaskingEngine instance. The library is written in C++, and this section is organized according to the namespace and class names used.
Standard implementation of IDataMaskingProfileContext and IDataMaskingProfileContextInspector tightly coupled to a specific version of a specific profile instance. It manages custom properties that are accepted by the associated profile, and rejects attempts to set those not accepted. The rejection is one way of letting the caller know that something it might deem essential is not handled by the profile.
Depends on CProfile, an abstract implementation of IDataMaskingProfile and IDataMaskingProfileInspector.
Implementation of IDataMaskingProfileMaskStyle providing customized masked output of values. Configuration options include:
@maximumVersion is an optional version number, needed only when the style does not apply to the maximum version of the value type in which it is defined.
@minimumVersion is an optional version number, needed only when the style does not apply to the minumum version of the value type in which it is defined.
@name is a required unique identifier for the style.
@overrideDefault is an optional Boolean flag controlling whether the style should replace the default style for the value type in which it is defined.
@pattern is an optional character sequence (one or more characters in length) that will be used to mask value content.
Am extension of CMaskStyle, this is intended to partially mask values, such as account numbers or telephone numbers, without assuming knowledge of the values. Configuration is generally expressed as:
@action what to do, either keep or mask (default); and
@location where to do it, either first or last (default); and
@count how many characters to examine, a positive integer value; and
@characters type of characters to be considered, either numbers, letters, alphanumeric, or all (default)
All four values may be omitted when only the inherited options are needed. If any of the four values are given, count is required and the other three are optional.
A value substring containing at most @count instances of the character class denoted by @characters is identified. Character classes are ASCII numeric characters (numbers), ASCII alphabetic characters (letters), ASCII alphabetic and numeric characters (alphanumeric), or any characters (all).
The value substring is at the start of the value when @location is first, and at the end of the value when @location is last.
The value substring is masked when @action is mask. All of the value with the exception of the substring is masked when @action is keep.
A base class for rule implementations, this defines standard rule properties without providing any content knowledge for use with maskContent.
Configuration includes:
@contentType is an optional, user-defined, label describing the content format to which the rule should be applied. maskContent requests that indicate a content type should apply all rules that match the type in addition to all rules that omit a type.
memberOf is an optional and repeating element identifying user-defined set labels. Only rules with membership in the requested set are considered; in the absence of an explictly requested set, a default set with an empty name is implied.
memberOf/@name is a required user-defined set label.
rule instances will frequently be specific to a content format. For example, a rule applied to XML markup value may include dependencies on XML syntax which are not applicable when masking JSON content. Rules specific to a markup format can be associated with that format using @contentType. Requests to mask content of a type will apply all rules without a @contentType value or with a matching value, and will exclude all rules with a non-matching value.
An extension of CRule, this identifies content substrings to be masked based on matching start and end tokens in the content buffer. For each occurrence of a configured start token that is balanced by a corresponding configured end token, the characters between the tokens are masked.
@endToken is an optional character sequence expected to immediately follow a content substring to be masked. This might be an XML element end tag, or a terminating double quote for a JSON value. A newline, i.e., \\n, is assumed if omitted or empty, for masking values such as HTTP headers.
@matchCase is an optional Boolean flag controlling whether token matches are case sensitive (true) or insensitivie (false). Matches are case insensitive by default. The implementation is based on ASCII data; case sensitive comparisons of similarly encoded non-ASCI text can work, but case insensitive comparisons are not supported.
@startToken is a required character sequence expected to precede a content substring to be masked.
No content type knowledge is implied by this class. An instance with @contentType of xml does not inherently know how to find values in XML markup. The defined tokens must include characters such as < and > to match an element name and quotes to match attribute values.
An implementation of IDataMaskingProfileIterator used for transforming a configuration property tree into a collection of profiles to be returned by a library entry point function. Created profiles are of the same class, identified by the template parameter profile_t.
Configuration includes:
profile An optional and repeating element defining a profile. The content of this element depends on profile_t.
If and only if profile is not included as a child of the configuration node, the node itself will be treated as a configuration for profile_t. The name of the configuration node is not required to be profile.
A concrete extension of CProfile, this manages value types and rules, providing a default implementation of maskValue but leaving other operations to subclasses.
Template parameters are:
valuetype_t identifies the concrete implementation of IDataMaskingProfileValueType to be instantiated during configuration.
rule_t identifies the concrete implementation of rules to be managed. Creation is assumed to be the responsibility of the value type.
context_t identifies the concrete implementation of IDataMaskingProfileContext to be instantiated on demand.
Configuration includes:
@defaultVersion is an optional version number identifying the version to be used when version 0 is requested. Omission or 0 implies the maximum version (which is configured before this value).
@domain is the required default identifier used to select this profile.
legacyDomain is an optional and repeating element identifying alternate identifiers by which the profile may be selected. This enables domain identifier naming conventions to be changed without breaking pre-existing references.
legacyDomain/@id is a required identifier of this profile.
@maximumVersion is an optional version number identifying the highest version supported by the configuration. Omission or 0 implies a value matching the minimum version (whether implicitly or explicitly defined).
@minimumVersion is an optional version number identifying the lowest version supported by the configuration. Omission or 0 implies 1.
@name is an optional label for the instance. If given, the value is used in trace output.
property is an optional and repeating element describing a custom context property recognized by the profile. The profile can declare awareness of properties without explicit use of them.
property/@name is a required context property name.
property/@minimumVersion is an optional version number indicating the lowest profile version aware of the property. Omission or 0 implies the profile minimum.
property/@maximumVersion is an optional version number indicating the highest profile version aware of the property. Omisssion or 0 implies the profile maximum.
valueType is an optional and repeating element describing the value types defined by the profile. Element content depends on the value of valuetype_t.
For profiles supporting a single version, value type names must be unique. For profiles supporting multiple versions, value type names may be repeated but must be unique for each version. To illustrate, consider this snippet:
In the example, foo and bar are each defined twice. The redefinition of foo is acceptable because each instance applies to a different version. The redefinition of bar is invalid because both instances claim to apply to version 2.
The value type name * is reserved by this class. A profile that includes a value type named * supports unconditional value masking. maskValue requests specifying unknown value type names can fall back to this special type and force masking for these values. Without this type, maskValue masks what it thinks should be masked; with this type, it trusts the caller to request masking for only those values known to require it. Value types included in unselected value type sets are known to the profile and, as such, can be used to prevent specific typed values from ever being masked.
The requirement of a type definition instead of a simpler flag is to enable the definition of mask styles. It has the side effect of enabling the definition of rules. In theory, an entire profile could be defined using a single value type. This may make sense in some cases, but not in all. For example, a partial mask style intended for use with U.S. Social Security numbers could be inappropriately applied to a password. Use care when configuring this type.
An extension of TProfile that adds support for maskContent. It is assumed by maskContent that each applicable rule must be applied serially, i.e., one after the other, using an bool applyRule(buffer, length, context) interface.
Template parameters are unchanged from TProfile.
Configuration options are unchanged from TProfile.
An implementation of IDataMaskingProfileValueType that manages mask styles and creates rules for the profile.
Template parameters are:
maskstyle_t identifies the concrete implementation of IDataMaskingProfileMaskStyle to be instantiated during configuration.
rule_t identifies the concrete implementation of rules to be created during configuration.
Configuration includes:
@maximumVersion is an optional version number, needed only when the type does not apply to the maximum version of the profile in which it is defined.
@minimumVersion is an optional version number, needed only when the type does not apply to the minumum version of the profile in which it is defined.
maskStyle is an optional and repeating element describing mask styles defined by the type. Element content depends on the value of maskstyle_t.
memberOf is an optional and repeating element identifying user-defined set labels. All value types that are not explicitly assigned set membership are considered for all masking requests. Value types assigned set membership are considered only when one of their assigned sets is selected with the request context.
memberOf/@name is a required user-defined set label.
@name is a required unique identifier for the type.
rule is an optional and repeating element describing rules defined by the type. Element content depends on the value of rule_t.
For value types supporting a single version, mask style names must be unique. For types supporting multiple versions, style names may be repeated but must be unique for each version. To illustrate, consider this snippet:
In the example, foo and bar are each defined twice. The redefinition of foo is acceptable because each instance applies to a different version. The redefinition of bar is invalid because both instances claim to apply to version 2.
This section describes the entry point functions exported by the shared library. The library must export one function, and may export multiple functions.
The description of each entry point will identify which of the previously described classes is used to represent the returned collection of profiles. If a templated class is identified, the template parameters are also listed. Refer to the class descriptions for additional information.
`,63)]))}const h=t(o,[["render",s]]);export{u as __pageData,h as default};
diff --git a/assets/system_security_plugins_jwtSecurity_README.md.B1LGJUHj.js b/assets/system_security_plugins_jwtSecurity_README.md.B1LGJUHj.js
new file mode 100644
index 00000000000..8ad890ce01f
--- /dev/null
+++ b/assets/system_security_plugins_jwtSecurity_README.md.B1LGJUHj.js
@@ -0,0 +1,8 @@
+import{_ as t,c as o,a3 as i,o as s}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"","description":"","frontmatter":{},"headers":[],"relativePath":"system/security/plugins/jwtSecurity/README.md","filePath":"system/security/plugins/jwtSecurity/README.md","lastUpdated":1731340314000}'),a={name:"system/security/plugins/jwtSecurity/README.md"};function n(r,e,l,d,c,h){return s(),o("div",null,e[0]||(e[0]=[i(`
The purpose of this plugin is to provide authentication and authorization capabilities for HPCC Systems users, with the credentials passed via valid JWT tokens.
The intention is to adhere as closely as possibly to the OpenID Connect (OIC) specification, which is a simple identity layer on top of the OAuth 2.0 protocol, while maintaining compatibility with the way HPCC Systems performs authentication and authorization today. More information about the OpenID Connect specification can be found at https://openid.net/specs/openid-connect-core-1_0.html.
One of the big advantages of OAuth 2.0 and OIC is that the service (in this case, HPCC Systems) never interacts with the user directly. Instead, authentication is performed by a trusted third party and the (successful) results are passed to the service in the form of a verifiable encoded token.
Unfortunately, HPCC Systems does not support the concept of third-party verification. It assumes that users -- really, any client application that operates as a user, including things like IDEs -- will submit username/password credentials for authentication. Until that is changed, HPCC Systems won't be able to fully adhere to the OIC specification.
We can, however, implement most of the specification. That is what this plugin does.
NOTE: This plugin is not available in a Windows build.
The plugin is called by the HPCC Systems esp process when a user needs to be authenticated. That call will contain the user's username and either a reference to a session token or a password. The session token is present only for already-authenticated users.
If the session token is not present, the plugin will call a JWT login service (also known as a JWT login endpoint) with the username and password, plus a nonce value for additional security.
That service authenticates the username/password credentials. If everything is good, the service constructs an OIC-compatible token that includes authorization information for that user and returns it to the plugin. The token is validated according to the OIC specification, including signature verification.
Note that token signature verification requires an additional piece of information. Tokens can be signed with a hash-based algorithm or with a public key-based algorithm (the actual algorithm used is determined by the JWT service). To verify either kind of algorithm, the plugin will need either the secret hash key or the public key that matches what the JWT service used. That key is read by the plugin from a file, and the file is determined by a configuration setting (see below). It is possible to change the contents of that file without restarting the esp process. Note, though, that the plugin may not notice that the file's contents have changed for several seconds (changes do not immediately take effect).
HPCC Systems uses a well-defined authorization scheme, originally designed around an LDAP implementation. That scheme is represented within the token as JWT claims. This plugin will unpack those claims and map to the authorization checks already in place within the HPCC Systems platform.
OIC includes the concept of refresh tokens. Refresh tokens enable a service to re-authorize an existing token without user intervention. Re-authorization typically happens due to a token expiring. Tokens should have a relatively short lifetime -- e.g. 15-30 minutes -- to promote good security and also give administrators the ability to modify a user's authorization while the user is logged in. This plugin fully supports refresh tokens by validating token lifetime at every authorization check and calling a JWT refresh service (also known as a JWT refresh endpoint) when needed. This largely follows the OIC specification.
Initial authentication: As stated above, the esp process will gather the username/password credentials instead of a third party, then send those credentials off to another service. In a true OIC configuration, the client process (the esp process) never sees user credentials and relies on an external service to gather them from the user.
The request made to the JWT login service is a POST HTTP or HTTPS call (depending on your configuration) containing four items in JSON format; example:
The most obvious outcome of this implementation is that a custom service/endpoint needs to be available. Or rather two services: One to handle the initial user login and one to handle token refreshes. Neither service precisely handles requests and replies in an OIC-compatible way, but the tokens themselves are OIC-compatible, which is good. That allows you to use third-party JWT libraries to construct and validate those tokens.
Several items must be defined in the platform's configuration. Within configmgr, the jwtsecmgr Security Manager plugin must be added as a component and then modified according to your environment:
The URL or unique name of this HPCC Systems cluster, used as the client_id in token requests
Full URL to the JWT Login Endpoint (should be HTTPS, but not required)
Full URL to the JWT Refresh Endpoint (should be HTTPS, but not required)
Boolean indicating whether to accept self-signed certificates for those endpoints; defaults to false
Secrets vault key/name or subdirectory under /opt/HPCCSystems/secrets/esp in which the JWT key used for the chosen signature algorithm is stored; defaults to "jwt-security"
Default permission access level (either "Full" or "None"); defaults to "Full"
Default workunit scope access level (either "Full" or "None"); defaults to "Full"
Default file scope access level (either "Full" or "None"); defaults to "Full"
Only the first three items have no default values and must be supplied.
Once the jwtseccmgr component is added, you have to tell other parts of the system to use the plugin. For user authentication and permissions affecting features and workunit scopes, you need to add the plugin to the esp component. Instructions for doing so can be found in the HPCC Systems Administrator's Guide manual (though the manual uses the htpasswd plugin as an example, the process is the same).
If you intend to implement file scope permissions then you will also need provide Dali information about the JWT plugin. In configmgr, within the Dali Server component, select the LDAP tab. Change the authMethod entry to secmgrPlugin and enter "jwtsecmgr" as the authPluginType. Make sure checkScopeScans is set to true.
This plugin supports all authorizations documented in the HPCC Systems® Administrator's Guide with the exception of "View Permissions". Loosely speaking, the permissions are divided into three groups: Feature, Workunit Scope, and File Scope.
Feature permissions are supported exactly as documented. A specific permission would exist as a JWT claim, by name, with the associated value being the name of the permission. For example, to grant read-only access to ECL Watch, use this claim:
{ "SmcAccess": "Read" }
File and workunit scope permissions are handled the same way, but different from feature permissions. The claim is one of the Claim constants in the tables below, and the associated value is a matching pattern. A pattern can be simple string or it can use wildcards (specifically, Linux's file globbing wildcards). Wildcards are not typically needed.
`,41)]))}const m=t(a,[["render",n]]);export{u as __pageData,m as default};
diff --git a/assets/system_security_plugins_jwtSecurity_README.md.B1LGJUHj.lean.js b/assets/system_security_plugins_jwtSecurity_README.md.B1LGJUHj.lean.js
new file mode 100644
index 00000000000..8ad890ce01f
--- /dev/null
+++ b/assets/system_security_plugins_jwtSecurity_README.md.B1LGJUHj.lean.js
@@ -0,0 +1,8 @@
+import{_ as t,c as o,a3 as i,o as s}from"./chunks/framework.DkhCEVKm.js";const u=JSON.parse('{"title":"","description":"","frontmatter":{},"headers":[],"relativePath":"system/security/plugins/jwtSecurity/README.md","filePath":"system/security/plugins/jwtSecurity/README.md","lastUpdated":1731340314000}'),a={name:"system/security/plugins/jwtSecurity/README.md"};function n(r,e,l,d,c,h){return s(),o("div",null,e[0]||(e[0]=[i(`
The purpose of this plugin is to provide authentication and authorization capabilities for HPCC Systems users, with the credentials passed via valid JWT tokens.
The intention is to adhere as closely as possibly to the OpenID Connect (OIC) specification, which is a simple identity layer on top of the OAuth 2.0 protocol, while maintaining compatibility with the way HPCC Systems performs authentication and authorization today. More information about the OpenID Connect specification can be found at https://openid.net/specs/openid-connect-core-1_0.html.
One of the big advantages of OAuth 2.0 and OIC is that the service (in this case, HPCC Systems) never interacts with the user directly. Instead, authentication is performed by a trusted third party and the (successful) results are passed to the service in the form of a verifiable encoded token.
Unfortunately, HPCC Systems does not support the concept of third-party verification. It assumes that users -- really, any client application that operates as a user, including things like IDEs -- will submit username/password credentials for authentication. Until that is changed, HPCC Systems won't be able to fully adhere to the OIC specification.
We can, however, implement most of the specification. That is what this plugin does.
NOTE: This plugin is not available in a Windows build.
The plugin is called by the HPCC Systems esp process when a user needs to be authenticated. That call will contain the user's username and either a reference to a session token or a password. The session token is present only for already-authenticated users.
If the session token is not present, the plugin will call a JWT login service (also known as a JWT login endpoint) with the username and password, plus a nonce value for additional security.
That service authenticates the username/password credentials. If everything is good, the service constructs an OIC-compatible token that includes authorization information for that user and returns it to the plugin. The token is validated according to the OIC specification, including signature verification.
Note that token signature verification requires an additional piece of information. Tokens can be signed with a hash-based algorithm or with a public key-based algorithm (the actual algorithm used is determined by the JWT service). To verify either kind of algorithm, the plugin will need either the secret hash key or the public key that matches what the JWT service used. That key is read by the plugin from a file, and the file is determined by a configuration setting (see below). It is possible to change the contents of that file without restarting the esp process. Note, though, that the plugin may not notice that the file's contents have changed for several seconds (changes do not immediately take effect).
HPCC Systems uses a well-defined authorization scheme, originally designed around an LDAP implementation. That scheme is represented within the token as JWT claims. This plugin will unpack those claims and map to the authorization checks already in place within the HPCC Systems platform.
OIC includes the concept of refresh tokens. Refresh tokens enable a service to re-authorize an existing token without user intervention. Re-authorization typically happens due to a token expiring. Tokens should have a relatively short lifetime -- e.g. 15-30 minutes -- to promote good security and also give administrators the ability to modify a user's authorization while the user is logged in. This plugin fully supports refresh tokens by validating token lifetime at every authorization check and calling a JWT refresh service (also known as a JWT refresh endpoint) when needed. This largely follows the OIC specification.
Initial authentication: As stated above, the esp process will gather the username/password credentials instead of a third party, then send those credentials off to another service. In a true OIC configuration, the client process (the esp process) never sees user credentials and relies on an external service to gather them from the user.
The request made to the JWT login service is a POST HTTP or HTTPS call (depending on your configuration) containing four items in JSON format; example:
The most obvious outcome of this implementation is that a custom service/endpoint needs to be available. Or rather two services: One to handle the initial user login and one to handle token refreshes. Neither service precisely handles requests and replies in an OIC-compatible way, but the tokens themselves are OIC-compatible, which is good. That allows you to use third-party JWT libraries to construct and validate those tokens.
Several items must be defined in the platform's configuration. Within configmgr, the jwtsecmgr Security Manager plugin must be added as a component and then modified according to your environment:
The URL or unique name of this HPCC Systems cluster, used as the client_id in token requests
Full URL to the JWT Login Endpoint (should be HTTPS, but not required)
Full URL to the JWT Refresh Endpoint (should be HTTPS, but not required)
Boolean indicating whether to accept self-signed certificates for those endpoints; defaults to false
Secrets vault key/name or subdirectory under /opt/HPCCSystems/secrets/esp in which the JWT key used for the chosen signature algorithm is stored; defaults to "jwt-security"
Default permission access level (either "Full" or "None"); defaults to "Full"
Default workunit scope access level (either "Full" or "None"); defaults to "Full"
Default file scope access level (either "Full" or "None"); defaults to "Full"
Only the first three items have no default values and must be supplied.
Once the jwtseccmgr component is added, you have to tell other parts of the system to use the plugin. For user authentication and permissions affecting features and workunit scopes, you need to add the plugin to the esp component. Instructions for doing so can be found in the HPCC Systems Administrator's Guide manual (though the manual uses the htpasswd plugin as an example, the process is the same).
If you intend to implement file scope permissions then you will also need provide Dali information about the JWT plugin. In configmgr, within the Dali Server component, select the LDAP tab. Change the authMethod entry to secmgrPlugin and enter "jwtsecmgr" as the authPluginType. Make sure checkScopeScans is set to true.
This plugin supports all authorizations documented in the HPCC Systems® Administrator's Guide with the exception of "View Permissions". Loosely speaking, the permissions are divided into three groups: Feature, Workunit Scope, and File Scope.
Feature permissions are supported exactly as documented. A specific permission would exist as a JWT claim, by name, with the associated value being the name of the permission. For example, to grant read-only access to ECL Watch, use this claim:
{ "SmcAccess": "Read" }
File and workunit scope permissions are handled the same way, but different from feature permissions. The claim is one of the Claim constants in the tables below, and the associated value is a matching pattern. A pattern can be simple string or it can use wildcards (specifically, Linux's file globbing wildcards). Wildcards are not typically needed.
`,41)]))}const m=t(a,[["render",n]]);export{u as __pageData,m as default};
diff --git a/assets/testing_regress_cleanupReadme.md.BWXfHXn3.js b/assets/testing_regress_cleanupReadme.md.BWXfHXn3.js
new file mode 100644
index 00000000000..bceac01f3a4
--- /dev/null
+++ b/assets/testing_regress_cleanupReadme.md.BWXfHXn3.js
@@ -0,0 +1 @@
+import{_ as t,c as a,a3 as r,o as s}from"./chunks/framework.DkhCEVKm.js";const m=JSON.parse('{"title":"Cleanup Parameter of Regression Suite run and query sub-command","description":"","frontmatter":{},"headers":[],"relativePath":"testing/regress/cleanupReadme.md","filePath":"testing/regress/cleanupReadme.md","lastUpdated":1731340314000}'),o={name:"testing/regress/cleanupReadme.md"};function i(n,e,l,u,d,c){return s(),a("div",null,e[0]||(e[0]=[r('
Cleanup Parameter of Regression Suite run and query sub-command
The cleanup parameter has been introduced to allow the user to automatically delete the workunits created by executing the Regression Suite on their local system.
It is an optional argument of the run and query sub-command.
A custom logging system also creates log files for each execution of the run and query sub-command that contains information about the workunit deletion.
[Pass] 1. Workunit Wuid=W20240526-094322 deleted successfully. [Pass] 2. Workunit Wuid=W20240526-094324 deleted successfully. [Failure] 3. Failed to delete Wuid=W20240526-094325. URL: http://127.0.0.1:8010/?Widget=WUDetailsWidget&Wuid=W20240526-094325 Failed cannot open workunit Wuid=W20240526-094325.. Response status code: 200 [Pass] 4. Workunit Wuid=W20240526-094327 deleted successfully. Suite destructor.
',15)]))}const h=t(o,[["render",i]]);export{m as __pageData,h as default};
diff --git a/assets/testing_regress_cleanupReadme.md.BWXfHXn3.lean.js b/assets/testing_regress_cleanupReadme.md.BWXfHXn3.lean.js
new file mode 100644
index 00000000000..bceac01f3a4
--- /dev/null
+++ b/assets/testing_regress_cleanupReadme.md.BWXfHXn3.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as a,a3 as r,o as s}from"./chunks/framework.DkhCEVKm.js";const m=JSON.parse('{"title":"Cleanup Parameter of Regression Suite run and query sub-command","description":"","frontmatter":{},"headers":[],"relativePath":"testing/regress/cleanupReadme.md","filePath":"testing/regress/cleanupReadme.md","lastUpdated":1731340314000}'),o={name:"testing/regress/cleanupReadme.md"};function i(n,e,l,u,d,c){return s(),a("div",null,e[0]||(e[0]=[r('
Cleanup Parameter of Regression Suite run and query sub-command
The cleanup parameter has been introduced to allow the user to automatically delete the workunits created by executing the Regression Suite on their local system.
It is an optional argument of the run and query sub-command.
A custom logging system also creates log files for each execution of the run and query sub-command that contains information about the workunit deletion.
[Pass] 1. Workunit Wuid=W20240526-094322 deleted successfully. [Pass] 2. Workunit Wuid=W20240526-094324 deleted successfully. [Failure] 3. Failed to delete Wuid=W20240526-094325. URL: http://127.0.0.1:8010/?Widget=WUDetailsWidget&Wuid=W20240526-094325 Failed cannot open workunit Wuid=W20240526-094325.. Response status code: 200 [Pass] 4. Workunit Wuid=W20240526-094327 deleted successfully. Suite destructor.
',15)]))}const h=t(o,[["render",i]]);export{m as __pageData,h as default};
diff --git a/assets/testing_regress_ecl_README.md.DWXau1FL.js b/assets/testing_regress_ecl_README.md.DWXau1FL.js
new file mode 100644
index 00000000000..45314c1a929
--- /dev/null
+++ b/assets/testing_regress_ecl_README.md.DWXau1FL.js
@@ -0,0 +1,64 @@
+import{_ as e,c as a,a3 as n,o as t}from"./chunks/framework.DkhCEVKm.js";const h=JSON.parse('{"title":"Test Suite for the Parquet Plugin","description":"","frontmatter":{},"headers":[],"relativePath":"testing/regress/ecl/README.md","filePath":"testing/regress/ecl/README.md","lastUpdated":1731340314000}'),i={name:"testing/regress/ecl/README.md"};function p(l,s,r,o,c,u){return t(),a("div",null,s[0]||(s[0]=[n(`
The Parquet plugin test suite is a subset of tests in the HPCC Systems regression suite. To run the tests:
Change directory to HPCC Platform/testing/regress.
To run the entire Parquet test suite:
Note: Some Parquet tests require initialization of Parquet files. Use the ./ecl-test setupcommand to initialize these files before running the test suite.
These commands can be run on any cluster, including hthor or Roxie like the example below.
This project focuses on the development of a comprehensive test suite for the recently integrated Parquet plugin within the HPCC Systems platform. The objective is to thoroughly evaluate the plugin's functionality, performance, and robustness across different scenarios and configurations. The key deliverables include defining and implementing various test cases, fixing any identified bugs, and providing extensive documentation.
The test suite will evaluate all data types supported by ECL and Arrow, as well as file operations, various compression formats, and schema handling. Additionally, the test suite will measure the plugin's performance across different HPCC components and hardware configurations, conduct stress tests to identify potential bottlenecks and bugs, and compare Parquet to other file formats used in the ecosystem, such as JSON, XML, and CSV.
Tests all available Arrow compression types: Snappy, GZip, Brotli, LZ4, ZSTD, Uncompressed Compares performance and file sizes for different compression options
The test suite generally uses key files located in HPCC-Platform/testing/regress/ecl/key, with a ".xml" extension, to evaluate test outcomes. These files store the expected results for comparison. However, some Parquet tests do not rely on key files, and alternative evaluation methods are used in those cases.
`,35)]))}const m=e(i,[["render",p]]);export{h as __pageData,m as default};
diff --git a/assets/testing_regress_ecl_README.md.DWXau1FL.lean.js b/assets/testing_regress_ecl_README.md.DWXau1FL.lean.js
new file mode 100644
index 00000000000..45314c1a929
--- /dev/null
+++ b/assets/testing_regress_ecl_README.md.DWXau1FL.lean.js
@@ -0,0 +1,64 @@
+import{_ as e,c as a,a3 as n,o as t}from"./chunks/framework.DkhCEVKm.js";const h=JSON.parse('{"title":"Test Suite for the Parquet Plugin","description":"","frontmatter":{},"headers":[],"relativePath":"testing/regress/ecl/README.md","filePath":"testing/regress/ecl/README.md","lastUpdated":1731340314000}'),i={name:"testing/regress/ecl/README.md"};function p(l,s,r,o,c,u){return t(),a("div",null,s[0]||(s[0]=[n(`
The Parquet plugin test suite is a subset of tests in the HPCC Systems regression suite. To run the tests:
Change directory to HPCC Platform/testing/regress.
To run the entire Parquet test suite:
Note: Some Parquet tests require initialization of Parquet files. Use the ./ecl-test setupcommand to initialize these files before running the test suite.
These commands can be run on any cluster, including hthor or Roxie like the example below.
This project focuses on the development of a comprehensive test suite for the recently integrated Parquet plugin within the HPCC Systems platform. The objective is to thoroughly evaluate the plugin's functionality, performance, and robustness across different scenarios and configurations. The key deliverables include defining and implementing various test cases, fixing any identified bugs, and providing extensive documentation.
The test suite will evaluate all data types supported by ECL and Arrow, as well as file operations, various compression formats, and schema handling. Additionally, the test suite will measure the plugin's performance across different HPCC components and hardware configurations, conduct stress tests to identify potential bottlenecks and bugs, and compare Parquet to other file formats used in the ecosystem, such as JSON, XML, and CSV.
Tests all available Arrow compression types: Snappy, GZip, Brotli, LZ4, ZSTD, Uncompressed Compares performance and file sizes for different compression options
The test suite generally uses key files located in HPCC-Platform/testing/regress/ecl/key, with a ".xml" extension, to evaluate test outcomes. These files store the expected results for comparison. However, some Parquet tests do not rely on key files, and alternative evaluation methods are used in those cases.
`,35)]))}const m=e(i,[["render",p]]);export{h as __pageData,m as default};
diff --git a/assets/testing_regress_ecl_analyzers_corporate_tmp_README.md.KZkrkEkE.js b/assets/testing_regress_ecl_analyzers_corporate_tmp_README.md.KZkrkEkE.js
new file mode 100644
index 00000000000..aa07087ee84
--- /dev/null
+++ b/assets/testing_regress_ecl_analyzers_corporate_tmp_README.md.KZkrkEkE.js
@@ -0,0 +1 @@
+import{_ as t,c as r,j as s,o as a}from"./chunks/framework.DkhCEVKm.js";const f=JSON.parse('{"title":"","description":"","frontmatter":{},"headers":[],"relativePath":"testing/regress/ecl/analyzers/corporate/tmp/README.md","filePath":"testing/regress/ecl/analyzers/corporate/tmp/README.md","lastUpdated":1731340314000}'),o={name:"testing/regress/ecl/analyzers/corporate/tmp/README.md"};function n(i,e,p,c,l,m){return a(),r("div",null,e[0]||(e[0]=[s("p",null,"This directory is for the function kbdumptree. The engine should create this but sometimes permission problems arrise.",-1)]))}const u=t(o,[["render",n]]);export{f as __pageData,u as default};
diff --git a/assets/testing_regress_ecl_analyzers_corporate_tmp_README.md.KZkrkEkE.lean.js b/assets/testing_regress_ecl_analyzers_corporate_tmp_README.md.KZkrkEkE.lean.js
new file mode 100644
index 00000000000..aa07087ee84
--- /dev/null
+++ b/assets/testing_regress_ecl_analyzers_corporate_tmp_README.md.KZkrkEkE.lean.js
@@ -0,0 +1 @@
+import{_ as t,c as r,j as s,o as a}from"./chunks/framework.DkhCEVKm.js";const f=JSON.parse('{"title":"","description":"","frontmatter":{},"headers":[],"relativePath":"testing/regress/ecl/analyzers/corporate/tmp/README.md","filePath":"testing/regress/ecl/analyzers/corporate/tmp/README.md","lastUpdated":1731340314000}'),o={name:"testing/regress/ecl/analyzers/corporate/tmp/README.md"};function n(i,e,p,c,l,m){return a(),r("div",null,e[0]||(e[0]=[s("p",null,"This directory is for the function kbdumptree. The engine should create this but sometimes permission problems arrise.",-1)]))}const u=t(o,[["render",n]]);export{f as __pageData,u as default};
diff --git a/assets/tools_esdlcmd_README.md.D3nMaAu6.js b/assets/tools_esdlcmd_README.md.D3nMaAu6.js
new file mode 100644
index 00000000000..0ccc7f07d89
--- /dev/null
+++ b/assets/tools_esdlcmd_README.md.D3nMaAu6.js
@@ -0,0 +1,307 @@
+import{_ as i,c as t,a3 as a,o as n}from"./chunks/framework.DkhCEVKm.js";const E=JSON.parse('{"title":"esdl Utility","description":"","frontmatter":{},"headers":[],"relativePath":"tools/esdlcmd/README.md","filePath":"tools/esdlcmd/README.md","lastUpdated":1731340314000}'),e={name:"tools/esdlcmd/README.md"};function l(h,s,p,k,o,d){return n(),t("div",null,s[0]||(s[0]=[a(`
The esdl utility tool aids with creating and managing ESDL-based and Dynamic ESDL services on an HPCC cluster. It consists of several different commands.
To generate output from an ESDL definition:
xml Generate XML from ESDL definition.
+ecl Generate ECL from ESDL definition.
+xsd Generate XSD from ESDL definition.
+wsdl Generate WSDL from ESDL definition.
+java Generate Java code from ESDL definition.
+cpp Generate C++ code from ESDL definition.
+monitor Generate ECL code for result monitoring / differencing
+monitor-template Generate a template for use with 'monitor' command
+
To manage ESDL and DESDL services:
publish Publish ESDL Definition for ESP use.
+list-definitions List all ESDL definitions.
+get-definition Get ESDL definition.
+delete Delete ESDL Definition.
+bind-service Configure ESDL based service on target ESP (with existing ESP Binding).
+list-bindings List all ESDL bindings.
+unbind-service Remove ESDL based service binding on target ESP.
+bind-method Configure method associated with existing ESDL binding.
+unbind-method Remove method associated with existing ESDL binding.
+get-binding Get ESDL binding.
+manifest Build a service binding or bundle from a manifest file.
+bind-log-transform Configure log transform associated with existing ESDL binding.
+unbind-log-transform Remove log transform associated with existing ESDL binding.
+
The sections below cover the commands in more detail.
The manifest command creates an XML configuration file for an ESDL ESP from an input XML manifest file. The type of configuration output depends on the manifest file input and on command-line options.
A manifest file is an XML-formatted template combining elements in and outside of the manifest's urn:hpcc:esdl:manifest namespace. Recognized elements of this namespace control the tool while all other markup is copied to the output. The goal of using a manifest file with the tool is to make configuring and deploying services easier:
The manifest file format abstracts some of the complexity of the actual configuration.
By allowing you to include external files like ESDL Scripts and XSLTs into the ouput, you can store and maintain them separately in your repo.
The result of running the manifest tool on a manifest file is an XML artifact suitable for use with the ESDL ESP. Supported output includes:
binding: The output is an ESDL binding that may be published to dali.
bundle: The output is an ESDL bundle file that may be used to launch an ESP in application mode.
The tool is permissive and flexible, copying through most markup to the output. Recognized elements in the manifest namespace may be treated differently. They are only required in order to take advantage of the automated processing and simplified format of the manifest file. This example highlights the recommended usage of manifest elements to use the tool's capabilities. Although you could replace some of the elements below with verbatim bundle or binding output elements we won't cover that usage here.
<em:Manifest> is the required root element. By default the tool outputs a bundle, though you may explicitly override that on the command line or by providing an @outputType='binding' attribute.
<em:ServiceBinding> is valid for both bundle and binding output. It is necessary to enable recognition of <em:Scripts>, and <em:Transform> elements.
<em:EsdlDefinition> is relevant only for bundle output. It is necessary to enable element order preservation and recognition of <em:Include> as a descendant element.
<em:Include> causes external file contents to be inserted in place of the element. The processing of included files is context dependent; the parent of the <em:Include> element dictates how the file is handled. File inclusion facilitates code reuse in a configuration as code development environment.
<em:Scripts> and <em:Transform> trigger preservation of element order for all descendent elements and enable <em:Include> recognition.
XML element ordering is significant to proper ESDL script, XSLT, and ESXDL content processing. The platform's IPropertyTree implementation, used when loading an artifact, does not preserve order. Output configuration files must embed order-sensitive content as text as opposed to XML markup. The tool allows configuration authors to create and maintain files as XML markup, which is easy to read. It then automates the conversion of the XML markup into the embedded text required by an ESP.
These elements may create artifact content, change the tool's behavior, or both. When used as intended, none of these elements will appear in the generated output:
Required root element of all manifest files that create output:
bundle output is created by default. It can be made explicit by setting either/or:
command line option --output-type to bundle
manifest property @outputType to bundle.
binding output is created when either/or:
command line option --output-type is binding
manifest property @outputType is binding.
Attribute
Required?
Value
Description
@outputType
N
string
A hint informing the tool which type of output to generate. The command line option --output-type may supersede this value to produce a different output. A bundle manifest is a superset of a binding manifest and may logically be used to create either output type. A binding manifest, as a subset of a bundle, cannot be used to create a valid bundle.
@xmlns[:prefix]
Y
string
The manifest namespace urn:hpcc:esdl:manifest must be declared. The default namespace prefix should not be used unless all other markup is fully qualified.
Recommended child of Manifest that creates ESDL binding content and applies ESDL binding-specific logic to descendent content:
A Binding output element is always created containing attributes of Manifest as required. See the sections below for details of the attributes referenced by the tool and how they're output.
A child of Binding named Definition is created with attributes @id and @esdlservice.
Recognition of <em:Scripts> elements is enabled.
Recognition of <em:Transform> elements is enabled.
While possible to omit the <em:ServiceBinding> element and instead embed a complete Binding tree in the manifest, it is discouraged because you lose the benefit of the processing described above.
There are three categories of attributes that can be defined on the <em:ServiceBinding> element:
Standard attributes needed for setup
Service-specific or binding-specific attributes
Auxillary attributes recommended for read-only reference
First are the standard attributes used to setup and define the binding:
Attribute
Required?
Value
Usage
@esdlservice
Yes
string
- Name of the ESDL service to which the binding is bound. Output on the Binding/Definition element. - Also used to generate a value for Definition/@id for bundle type output.
Note that Definition/@id is not output by the tool for binding output as it would need to match the ESDLDefinitionId passed on command line esdl bind-service call, which may differ between environments.
Second are binding-specific or service-specific attributes. This is an open-ended category where most future attributes will belong. Any attribute not in the other two categories is included here and output on the Binding/Definition element:
Attribute
Required?
Value
Usage
@auth_feature
No
string
Used declare authorization settings if they aren't present in the ESDL Definition, or override them if they are. Additional documentation on this attribute is forthcoming.
@returnSchemaLocationOnOK
No
Boolean
When true, a successful SOAP response (non SOAP-Fault) will include the schema location property. False by default.
@namespace
No
string
String specifying the namespace for all methods in the binding. May contain variables that are replaced with their values by the ESP during runtime: - \${service} : lowercase service name - \${esdl-service} : service name, possibly mixed case - \${method} : lowercase method name - \${esdl-method} : method name, possibly mixed case - \${optionals} : comma-delimited list of all optional URL parameters included in the method request, enclosed in parentheses - \${version} : client version number
Finally are auxillary attributes. These should be thought of as read-only or for reference, and it is not recommended that you set these in the manifest. They are set by the system when publishing to dali using esdl bind-service, and they are present on binding configurations retrieved from the dali. If you set these attributes in the manifest, the values will be overwritten when running esdl bind-service. Alternately, if you run as an esdl application, these values aren't set by default and don't have a material effect on the binding, but they may appear in the trace log.
Attribute
Required?
Value
Usage
@created
No
string
Timestamp of binding creation
@espbinding
No
string
Set to match the id. Otherwise the value is unset and unused.
@espprocess
No
string
Name of the ESP process this binding is running on
@id
No
string
Runtime name of the binding. When publishing to dali the value is [ESP Process].[port].[ESDL Service]. When not present in the manifest a default value is generated of the form [@esdlservice]_desdl_binding
@port
No
string
Port on the ESP Process listening for connections to the binding.
Optional element that imports the contents of another file into the output in place of itself. The outcome of the import depends on the context in which this element is used. See EsdlDefinition, Scripts, and Transform for more information.
Attribute
Required?
Value
Usage
@file
Yes
file path
Full or partial path to an external file to be imported. If a partial path is outside of the tool's working directory, the tool's command line must specify the appropriate root directory using either -I or --include-path.
Any XSLTs or ESDL Scripts written inline in a manifest file will have XML escaping applied where required to generate valid XML. If an XSLT contains any text content or markup that needs to be preserved as-is (no XML escaping applied) then be sure to use an <em:Include> operation. Included files are inserted into the output as-is, with the exception of encoding nested CDATA markup. If the included file will be inside a CDATA section on output, then any CDATA end markup in the file will be encoded as ]]]]><![CDATA[> to prevent nested CDATA sections or a prematurely ending a CDATA section.
Optional repeatable element appearing within an <em:ServiceBinding> element that processes child elements and creates output expected for an ESDL binding:
The <em:Scripts> element is replaced on output with <Scripts>. Then all content is enclosed in a CDATA section after wrapping it with a new Scripts element. That new <Scripts> element contains namespaces declared by the input <em:Scripts> element. The input <em:Scripts foo="..." xmlns:bar="..."><!-- content --></em:Scripts> becomes <Scripts><![CDATA[<Scripts xmlns:bar="..."><!-- content --></Scripts>]]></Scripts>.
The manifest <em:Include> element is recognized to import scripts from external files. The entire file, minus leading and trailing whitespace, is imported. Refrain from including files that contain an XML declaration.
Optional repeatable element appearing within an <em:ServiceBinding> element that processes child elements and creates output expected for an ESDL binding:
All content is enclosed in a CDATA section. The input <em:Transform> <!-- content --> </em:Transform> becomes <Transform><![CDATA[<!-- content -->]]></Transform>.
The manifest <em:Include> element is recognized to import transforms from external files. The entire file, minus leading and trailing whitespace, is imported. Refrain from including files that contain an XML declaration.
Usage:
+
+esdl manifest <manifest-file> [options]
+
+Options:
+ -I | --include-path <path>
+ Search path for external files included in the manifest.
+ Use once for each path.
+ --outfile <filename>
+ Path and name of the output file
+ --output-type <type>
+ When specified this option overrides the value supplied
+ in the manifest attribute Manifest/@outputType.
+ Allowed values are 'binding' or 'bundle'.
+ When not specified in either location the default is
+ 'bundle'
+ --help Display usage information for the given command
+ -v,--verbose Output additional tracing information
+ -tcat,--trace-category <flags>
+ Control which debug messages are output; a case-insensitive
+ comma-delimited combination of:
+ dev: all output for the developer audience
+ admin: all output for the operator audience
+ user: all output for the user audience
+ err: all error output
+ warn: all warning output
+ prog: all progress output
+ info: all info output
+ Errors and warnings are enabled by default if not verbose,
+ and all are enabled when verbose. Use an empty <flags> value
+ to disable all.
+
The esdl manifest command reads the manifest, processes statements in the urn:hpcc:esdl:manifest namespace and generates an output XML file formatted to the requirements of the ESDL ESP. This includes wrapping included content in CDATA sections to ensure element order is maintained and replacing urn:hpcc:esdl:manifest elements as required.
An example output of each type -bundle and binding- is shown below. The examples use the sample manifest above as input plus these included files:
`,78)]))}const g=i(e,[["render",l]]);export{E as __pageData,g as default};
diff --git a/assets/tools_esdlcmd_README.md.D3nMaAu6.lean.js b/assets/tools_esdlcmd_README.md.D3nMaAu6.lean.js
new file mode 100644
index 00000000000..0ccc7f07d89
--- /dev/null
+++ b/assets/tools_esdlcmd_README.md.D3nMaAu6.lean.js
@@ -0,0 +1,307 @@
+import{_ as i,c as t,a3 as a,o as n}from"./chunks/framework.DkhCEVKm.js";const E=JSON.parse('{"title":"esdl Utility","description":"","frontmatter":{},"headers":[],"relativePath":"tools/esdlcmd/README.md","filePath":"tools/esdlcmd/README.md","lastUpdated":1731340314000}'),e={name:"tools/esdlcmd/README.md"};function l(h,s,p,k,o,d){return n(),t("div",null,s[0]||(s[0]=[a(`
The esdl utility tool aids with creating and managing ESDL-based and Dynamic ESDL services on an HPCC cluster. It consists of several different commands.
To generate output from an ESDL definition:
xml Generate XML from ESDL definition.
+ecl Generate ECL from ESDL definition.
+xsd Generate XSD from ESDL definition.
+wsdl Generate WSDL from ESDL definition.
+java Generate Java code from ESDL definition.
+cpp Generate C++ code from ESDL definition.
+monitor Generate ECL code for result monitoring / differencing
+monitor-template Generate a template for use with 'monitor' command
+
To manage ESDL and DESDL services:
publish Publish ESDL Definition for ESP use.
+list-definitions List all ESDL definitions.
+get-definition Get ESDL definition.
+delete Delete ESDL Definition.
+bind-service Configure ESDL based service on target ESP (with existing ESP Binding).
+list-bindings List all ESDL bindings.
+unbind-service Remove ESDL based service binding on target ESP.
+bind-method Configure method associated with existing ESDL binding.
+unbind-method Remove method associated with existing ESDL binding.
+get-binding Get ESDL binding.
+manifest Build a service binding or bundle from a manifest file.
+bind-log-transform Configure log transform associated with existing ESDL binding.
+unbind-log-transform Remove log transform associated with existing ESDL binding.
+
The sections below cover the commands in more detail.
The manifest command creates an XML configuration file for an ESDL ESP from an input XML manifest file. The type of configuration output depends on the manifest file input and on command-line options.
A manifest file is an XML-formatted template combining elements in and outside of the manifest's urn:hpcc:esdl:manifest namespace. Recognized elements of this namespace control the tool while all other markup is copied to the output. The goal of using a manifest file with the tool is to make configuring and deploying services easier:
The manifest file format abstracts some of the complexity of the actual configuration.
By allowing you to include external files like ESDL Scripts and XSLTs into the ouput, you can store and maintain them separately in your repo.
The result of running the manifest tool on a manifest file is an XML artifact suitable for use with the ESDL ESP. Supported output includes:
binding: The output is an ESDL binding that may be published to dali.
bundle: The output is an ESDL bundle file that may be used to launch an ESP in application mode.
The tool is permissive and flexible, copying through most markup to the output. Recognized elements in the manifest namespace may be treated differently. They are only required in order to take advantage of the automated processing and simplified format of the manifest file. This example highlights the recommended usage of manifest elements to use the tool's capabilities. Although you could replace some of the elements below with verbatim bundle or binding output elements we won't cover that usage here.
<em:Manifest> is the required root element. By default the tool outputs a bundle, though you may explicitly override that on the command line or by providing an @outputType='binding' attribute.
<em:ServiceBinding> is valid for both bundle and binding output. It is necessary to enable recognition of <em:Scripts>, and <em:Transform> elements.
<em:EsdlDefinition> is relevant only for bundle output. It is necessary to enable element order preservation and recognition of <em:Include> as a descendant element.
<em:Include> causes external file contents to be inserted in place of the element. The processing of included files is context dependent; the parent of the <em:Include> element dictates how the file is handled. File inclusion facilitates code reuse in a configuration as code development environment.
<em:Scripts> and <em:Transform> trigger preservation of element order for all descendent elements and enable <em:Include> recognition.
XML element ordering is significant to proper ESDL script, XSLT, and ESXDL content processing. The platform's IPropertyTree implementation, used when loading an artifact, does not preserve order. Output configuration files must embed order-sensitive content as text as opposed to XML markup. The tool allows configuration authors to create and maintain files as XML markup, which is easy to read. It then automates the conversion of the XML markup into the embedded text required by an ESP.
These elements may create artifact content, change the tool's behavior, or both. When used as intended, none of these elements will appear in the generated output:
Required root element of all manifest files that create output:
bundle output is created by default. It can be made explicit by setting either/or:
command line option --output-type to bundle
manifest property @outputType to bundle.
binding output is created when either/or:
command line option --output-type is binding
manifest property @outputType is binding.
Attribute
Required?
Value
Description
@outputType
N
string
A hint informing the tool which type of output to generate. The command line option --output-type may supersede this value to produce a different output. A bundle manifest is a superset of a binding manifest and may logically be used to create either output type. A binding manifest, as a subset of a bundle, cannot be used to create a valid bundle.
@xmlns[:prefix]
Y
string
The manifest namespace urn:hpcc:esdl:manifest must be declared. The default namespace prefix should not be used unless all other markup is fully qualified.
Recommended child of Manifest that creates ESDL binding content and applies ESDL binding-specific logic to descendent content:
A Binding output element is always created containing attributes of Manifest as required. See the sections below for details of the attributes referenced by the tool and how they're output.
A child of Binding named Definition is created with attributes @id and @esdlservice.
Recognition of <em:Scripts> elements is enabled.
Recognition of <em:Transform> elements is enabled.
While possible to omit the <em:ServiceBinding> element and instead embed a complete Binding tree in the manifest, it is discouraged because you lose the benefit of the processing described above.
There are three categories of attributes that can be defined on the <em:ServiceBinding> element:
Standard attributes needed for setup
Service-specific or binding-specific attributes
Auxillary attributes recommended for read-only reference
First are the standard attributes used to setup and define the binding:
Attribute
Required?
Value
Usage
@esdlservice
Yes
string
- Name of the ESDL service to which the binding is bound. Output on the Binding/Definition element. - Also used to generate a value for Definition/@id for bundle type output.
Note that Definition/@id is not output by the tool for binding output as it would need to match the ESDLDefinitionId passed on command line esdl bind-service call, which may differ between environments.
Second are binding-specific or service-specific attributes. This is an open-ended category where most future attributes will belong. Any attribute not in the other two categories is included here and output on the Binding/Definition element:
Attribute
Required?
Value
Usage
@auth_feature
No
string
Used declare authorization settings if they aren't present in the ESDL Definition, or override them if they are. Additional documentation on this attribute is forthcoming.
@returnSchemaLocationOnOK
No
Boolean
When true, a successful SOAP response (non SOAP-Fault) will include the schema location property. False by default.
@namespace
No
string
String specifying the namespace for all methods in the binding. May contain variables that are replaced with their values by the ESP during runtime: - \${service} : lowercase service name - \${esdl-service} : service name, possibly mixed case - \${method} : lowercase method name - \${esdl-method} : method name, possibly mixed case - \${optionals} : comma-delimited list of all optional URL parameters included in the method request, enclosed in parentheses - \${version} : client version number
Finally are auxillary attributes. These should be thought of as read-only or for reference, and it is not recommended that you set these in the manifest. They are set by the system when publishing to dali using esdl bind-service, and they are present on binding configurations retrieved from the dali. If you set these attributes in the manifest, the values will be overwritten when running esdl bind-service. Alternately, if you run as an esdl application, these values aren't set by default and don't have a material effect on the binding, but they may appear in the trace log.
Attribute
Required?
Value
Usage
@created
No
string
Timestamp of binding creation
@espbinding
No
string
Set to match the id. Otherwise the value is unset and unused.
@espprocess
No
string
Name of the ESP process this binding is running on
@id
No
string
Runtime name of the binding. When publishing to dali the value is [ESP Process].[port].[ESDL Service]. When not present in the manifest a default value is generated of the form [@esdlservice]_desdl_binding
@port
No
string
Port on the ESP Process listening for connections to the binding.
Optional element that imports the contents of another file into the output in place of itself. The outcome of the import depends on the context in which this element is used. See EsdlDefinition, Scripts, and Transform for more information.
Attribute
Required?
Value
Usage
@file
Yes
file path
Full or partial path to an external file to be imported. If a partial path is outside of the tool's working directory, the tool's command line must specify the appropriate root directory using either -I or --include-path.
Any XSLTs or ESDL Scripts written inline in a manifest file will have XML escaping applied where required to generate valid XML. If an XSLT contains any text content or markup that needs to be preserved as-is (no XML escaping applied) then be sure to use an <em:Include> operation. Included files are inserted into the output as-is, with the exception of encoding nested CDATA markup. If the included file will be inside a CDATA section on output, then any CDATA end markup in the file will be encoded as ]]]]><![CDATA[> to prevent nested CDATA sections or a prematurely ending a CDATA section.
Optional repeatable element appearing within an <em:ServiceBinding> element that processes child elements and creates output expected for an ESDL binding:
The <em:Scripts> element is replaced on output with <Scripts>. Then all content is enclosed in a CDATA section after wrapping it with a new Scripts element. That new <Scripts> element contains namespaces declared by the input <em:Scripts> element. The input <em:Scripts foo="..." xmlns:bar="..."><!-- content --></em:Scripts> becomes <Scripts><![CDATA[<Scripts xmlns:bar="..."><!-- content --></Scripts>]]></Scripts>.
The manifest <em:Include> element is recognized to import scripts from external files. The entire file, minus leading and trailing whitespace, is imported. Refrain from including files that contain an XML declaration.
Optional repeatable element appearing within an <em:ServiceBinding> element that processes child elements and creates output expected for an ESDL binding:
All content is enclosed in a CDATA section. The input <em:Transform> <!-- content --> </em:Transform> becomes <Transform><![CDATA[<!-- content -->]]></Transform>.
The manifest <em:Include> element is recognized to import transforms from external files. The entire file, minus leading and trailing whitespace, is imported. Refrain from including files that contain an XML declaration.
Usage:
+
+esdl manifest <manifest-file> [options]
+
+Options:
+ -I | --include-path <path>
+ Search path for external files included in the manifest.
+ Use once for each path.
+ --outfile <filename>
+ Path and name of the output file
+ --output-type <type>
+ When specified this option overrides the value supplied
+ in the manifest attribute Manifest/@outputType.
+ Allowed values are 'binding' or 'bundle'.
+ When not specified in either location the default is
+ 'bundle'
+ --help Display usage information for the given command
+ -v,--verbose Output additional tracing information
+ -tcat,--trace-category <flags>
+ Control which debug messages are output; a case-insensitive
+ comma-delimited combination of:
+ dev: all output for the developer audience
+ admin: all output for the operator audience
+ user: all output for the user audience
+ err: all error output
+ warn: all warning output
+ prog: all progress output
+ info: all info output
+ Errors and warnings are enabled by default if not verbose,
+ and all are enabled when verbose. Use an empty <flags> value
+ to disable all.
+
The esdl manifest command reads the manifest, processes statements in the urn:hpcc:esdl:manifest namespace and generates an output XML file formatted to the requirements of the ESDL ESP. This includes wrapping included content in CDATA sections to ensure element order is maintained and replacing urn:hpcc:esdl:manifest elements as required.
An example output of each type -bundle and binding- is shown below. The examples use the sample manifest above as input plus these included files:
`,78)]))}const g=i(e,[["render",l]]);export{E as __pageData,g as default};
diff --git a/assets/tools_esp-api_README.md.WcTRZBDR.js b/assets/tools_esp-api_README.md.WcTRZBDR.js
new file mode 100644
index 00000000000..5cc0960cf54
--- /dev/null
+++ b/assets/tools_esp-api_README.md.WcTRZBDR.js
@@ -0,0 +1 @@
+import{_ as o,c as t,a3 as s,o as r}from"./chunks/framework.DkhCEVKm.js";const h=JSON.parse('{"title":"Developer README for ESP API Command Line Tool","description":"","frontmatter":{},"headers":[],"relativePath":"tools/esp-api/README.md","filePath":"tools/esp-api/README.md","lastUpdated":1731340314000}'),a={name:"tools/esp-api/README.md"};function i(n,e,l,d,p,c){return r(),t("div",null,e[0]||(e[0]=[s('
ESDL Directory Location: The tool gathers the directory of the ESDL files from an environment configuration variable; gets the install path from jutil library and appends /componentfiles/esdl_files/.
Server and Port Defaults: When using the test command, if the server and port are not specified, the tool defaults to interacting with an ESP instance located at http://127.0.0.1:8010.
Custom ESDL Directory Argument: An additional argument could be introduced to allow users to specify the directory of the ESDL files directly in the command.
Template Request Generation: A feature could be added to generate template XML or JSON requests. This would simplify the process of filling out requests by providing a pre-structured template.
Credential Prompts: The tool could be expanded to prompt for the username and password upon a 401 Unauthorized response.
Selective Response Extraction: Another potential feature is to allow extraction of specific tags from the response using XPath expressions. This would make it easier to parse and analyze responses.
',8)]))}const u=o(a,[["render",i]]);export{h as __pageData,u as default};
diff --git a/assets/tools_esp-api_README.md.WcTRZBDR.lean.js b/assets/tools_esp-api_README.md.WcTRZBDR.lean.js
new file mode 100644
index 00000000000..5cc0960cf54
--- /dev/null
+++ b/assets/tools_esp-api_README.md.WcTRZBDR.lean.js
@@ -0,0 +1 @@
+import{_ as o,c as t,a3 as s,o as r}from"./chunks/framework.DkhCEVKm.js";const h=JSON.parse('{"title":"Developer README for ESP API Command Line Tool","description":"","frontmatter":{},"headers":[],"relativePath":"tools/esp-api/README.md","filePath":"tools/esp-api/README.md","lastUpdated":1731340314000}'),a={name:"tools/esp-api/README.md"};function i(n,e,l,d,p,c){return r(),t("div",null,e[0]||(e[0]=[s('
ESDL Directory Location: The tool gathers the directory of the ESDL files from an environment configuration variable; gets the install path from jutil library and appends /componentfiles/esdl_files/.
Server and Port Defaults: When using the test command, if the server and port are not specified, the tool defaults to interacting with an ESP instance located at http://127.0.0.1:8010.
Custom ESDL Directory Argument: An additional argument could be introduced to allow users to specify the directory of the ESDL files directly in the command.
Template Request Generation: A feature could be added to generate template XML or JSON requests. This would simplify the process of filling out requests by providing a pre-structured template.
Credential Prompts: The tool could be expanded to prompt for the username and password upon a 401 Unauthorized response.
Selective Response Extraction: Another potential feature is to allow extraction of specific tags from the response using XPath expressions. This would make it easier to parse and analyze responses.
',8)]))}const u=o(a,[["render",i]]);export{h as __pageData,u as default};
diff --git a/assets/tools_tagging_README.md.CdzS0CPr.js b/assets/tools_tagging_README.md.CdzS0CPr.js
new file mode 100644
index 00000000000..75904aa1764
--- /dev/null
+++ b/assets/tools_tagging_README.md.CdzS0CPr.js
@@ -0,0 +1,13 @@
+import{_ as a,c as s,a3 as n,o as i}from"./chunks/framework.DkhCEVKm.js";const d=JSON.parse('{"title":"Tagging new versions","description":"","frontmatter":{},"headers":[],"relativePath":"tools/tagging/README.md","filePath":"tools/tagging/README.md","lastUpdated":1731340314000}'),t={name:"tools/tagging/README.md"};function o(l,e,p,r,c,g){return i(),s("div",null,e[0]||(e[0]=[n(`
The following process should be followed when tagging a new set of versions.
Upmerge all changes between candidate branches for the different versions
You can set the all environment variable to a subset of the projects (e.g. export all=hpcc) if there are no changes in the other repositories. The only effect for projects that are upmerged with no changes will be that they gain an empty merge transaction. If multiple people are merging PRs to different repositories it may be safer to upmerge all projects.
If you have merged changes onto a point-release branch you would normally create a new rc before going gold. If the change was trivial (e.g. removing an unwanted file) then you can use the --ignore option to skip that step.
Creating a new rc for an existing point release:
This normally happens after cherry-picking a late fix for a particular version, which has already been merged into the .x candidate branch.
A new minor branch is created from the current master...
./gominor.sh
`,33)]))}const u=a(t,[["render",o]]);export{d as __pageData,u as default};
diff --git a/assets/tools_tagging_README.md.CdzS0CPr.lean.js b/assets/tools_tagging_README.md.CdzS0CPr.lean.js
new file mode 100644
index 00000000000..75904aa1764
--- /dev/null
+++ b/assets/tools_tagging_README.md.CdzS0CPr.lean.js
@@ -0,0 +1,13 @@
+import{_ as a,c as s,a3 as n,o as i}from"./chunks/framework.DkhCEVKm.js";const d=JSON.parse('{"title":"Tagging new versions","description":"","frontmatter":{},"headers":[],"relativePath":"tools/tagging/README.md","filePath":"tools/tagging/README.md","lastUpdated":1731340314000}'),t={name:"tools/tagging/README.md"};function o(l,e,p,r,c,g){return i(),s("div",null,e[0]||(e[0]=[n(`
The following process should be followed when tagging a new set of versions.
Upmerge all changes between candidate branches for the different versions
You can set the all environment variable to a subset of the projects (e.g. export all=hpcc) if there are no changes in the other repositories. The only effect for projects that are upmerged with no changes will be that they gain an empty merge transaction. If multiple people are merging PRs to different repositories it may be safer to upmerge all projects.
If you have merged changes onto a point-release branch you would normally create a new rc before going gold. If the change was trivial (e.g. removing an unwanted file) then you can use the --ignore option to skip that step.
Creating a new rc for an existing point release:
This normally happens after cherry-picking a late fix for a particular version, which has already been merged into the .x candidate branch.
: - CMakeLists.txt - Root CMake file - version.cmake - common cmake file where version variables are set - build-config.h.cmake - cmake generation template for build-config.h
\- cmake\_modules/ - Directory storing modules and configurations for CMake
+
+: - FindXXXXX.cmake - CMake find files used to locate libraries,
+ headers, and binaries
+ - commonSetup.cmake - common configuration settings for the
+ entire project (contains configure time options)
+ - docMacros.cmake - common documentation macros used for
+ generating fop and pdf files
+ - optionDefaults.cmake - contains common variables for the
+ platform build
+ - distrocheck.sh - script that determines if the OS uses DEB
+ or RPM
+ - getpackagerevisionarch.sh - script that returns OS version
+ and arch in format used for packaging
+
+ \- dependencies/ - Directory storing dependency files used for package dependencies
+
+ : - \<OS\>.cmake - File containing either DEB or RPM
+ dependencies for the given OS
+
+\- build-utils/ - Directory for build related utilities
+
+: - cleanDeb.sh - script that unpacks a deb file and rebuilds
+ with fakeroot to clean up lintain errors/warnings
+
set_source_files_properties(<file> PROPERTIES GENERATED TRUE) - Must be set on generated source files or dependency generation fails and increases build time.
Using custom commands between multiple cmake files
GET_TARGET_PROPERTY(<VAR from other cmake file> <var for this file> LOCATION)
GET_TARGET_PROPERTY(ESDL_EXE esdl LOCATION) - will get from the top level cache the ESDL_EXE value and set it in esdl for your current cmake file
USE add_custom_command only when 100% needed.
All directories in a cmake project should have a CMakeLists.txt file and be called from the upper level project with an add_subdirectory or HPCC_ADD_SUBDIRECTORY
When you have a property that will be shared between cmake projects use define_property to set it in the top level cache.
define_property(GLOBAL PROPERTY TEST_TARGET BRIEF_DOCS "test doc" FULL_DOCS "Full test doc")
mark_as_advanced(TEST_TARGET) - this is required to force the property into the top level cache.CMake Layout:
NOT XXXXX_FOUND
+ Externals set
+ define needed vars for finding external based libraries/headers
+
+ Use Native set
+ use FIND_PATH to locate headers
+ use FIND_LIBRARY to find libs
+
+Include Cmake macros file for package handling
+define package handling args for find return (This will set XXXXX_FOUND)
+
+XXXXX_FOUND
+ perform any modifications you feel is needed for the find
+
+Mark defined variables used in package handling args as advanced for return
+
Will define when done:
XXXXX_FOUND
+XXXXX_INCLUDE_DIR
+XXXXX_LIBRARIES
+
(more can be defined, but must be at min the previous unless looking for only a binary)
For an example, see FindAPR.cmake
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/CodeGenerator.html b/devdoc/CodeGenerator.html
new file mode 100644
index 00000000000..10798fa75b1
--- /dev/null
+++ b/devdoc/CodeGenerator.html
@@ -0,0 +1,36 @@
+
+
+
+
+
+ Eclcc/Code generator | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
The code generator has to do its job accurately. If the code generator does not correctly map the ECL to the workunit it can lead to corrupt data and invalid results. Problems like that can often be very hard and frustrating for the ECL users to track down.
There is also a strong emphasis on generating output that is as good as possible. Eclcc contains many different optimization stages, and is extensible to allow others to be easily added.
Eclcc needs to be able to cope with reasonably large jobs. Queries that contain several megabytes of ECL, and generate tens of thousands of activities, and 10s of Mb of C++ are routine. These queries need to be processed relatively quickly.
Nearly all the processing of ECL is done using an expression graph. The representation of the expression graph has some particular characteristics:
Once the nodes in the expression graph have been created they are NEVER modified.
Nodes in the expression graph are ALWAYS commoned up if they are identical.
Each node in the expression graph is link counted (see below) to track its lifetime.
If a modified graph is required a new graph is created (sharing nodes from the old one)
The ECL language is a declarative language, and in general is assumed to be pure - i.e. there are no side-effects, expressions can be evaluated lazily and re-evaluating an expression causes no problems. This allows eclcc to transform the graph in lots of interesting ways. (Life is never that simple so there are mechanisms for handling the exceptions.)
One of the main challenges with eclcc is converting the declarative ECL code into imperative C++ code. One key problem is it needs to try to ensure that code is only evaluated when it is required, but that it is also only evaluated once. It isn't always possible to satisfy both constraints - for example a global dataset expression used within a child query. Should it be evaluated once before the activity containing the child query is called, or each time the child query is called? If it is called on demand then it may not be evaluated as efficiently...
This issue complicates many of the optimizations and transformations that are done to the queries. Long term the plan is to allow the engines to support more delayed lazy-evaluation, so that whether something is evaluated is more dynamic rather than static.
The idealised view of the processing within eclcc follows the following stages:
Parse the ECL into an expression graph.
Expand out function calls.
Normalize the expression graph so it is in a consistent format.
Normalize the references to fields within datasets to tie them up with their scopes.
Apply various global optimizations.
Translate logical operations into the activities that will implement them.
Resource and generate the global graph
For each activity, resource, optimize and generate its child graphs.
In practice the progression is not so clear cut. There tends to be some overlap between the different stages, and some of them may occur in slightly different orders. However the order broadly holds.
Before any change is accepted for the code generator it is always run against several regression suites to ensure that it doesn't introduce any problems, and that the change has the desired effect. There are several different regression suites:
testing/regress/ecl - The run time regression suite.
ecl/regress - a compiler regression suite. This contains tests that cannot run and error tests.
LN private suite - This contains a large selection (>10Gb) of archived queries. The contain proprietary code so unfortunately cannot be released as open source.
The ecl/regress directory contains a script 'regress.sh' that is used for running the regression tests. It should be executed in the directory containing the ecl files. The script generates the c++ code (and workunits) for each of the source files to a target directory, and then executes a comparison program to compare the new results with a previous "golden" reference set.
Before making any changes to the compiler, a reference set should be created by running the regression script and copying the generated files to the reference directory.
(A version of this command resides in a shell script in each of my regression suite directories, with the -t and -c options adapted for each suite.)
For a full list of options execute the script with no parameters, or take a look at the script itself. A couple of useful options are:
The script can be run on a single file by using the -q option.
The (-e) option selects the path of the eclcc. This is particularly useful when running from the build directory (see below), or using multiple build directories to compare behaviour between different versions.
We strongly recommend using a comparison program which allows rules to be defined to ignore certain differences (e.g., beyond compare).
It is much quicker to run eclcc directly from the build directory, rather than deploying a system and running eclcc from there. To do this you need to configure some options that eclcc requires, e.g. where the include files are found. The options can be set by either setting environment variables or by specifiying options in an eclcc.ini file. The following are the names of the different options:
The eclcc.ini can either be a file in the local directory, or specified on the eclcc command line with -specs. Including the settings in a local eclcc.ini file also it easy to debug eclcc directly from the build directory within the eclipse environment.
There is an option for eclcc to output a logging file, and another to specify the level of detail in that logging file. If the detail level is above 500 then the expresssion tree for the query is output to the logging file after each of the code transformations. The tracing is very useful for tracking down at which stage inconsistencies are introduced in the expression graph, and also for learning how each transformation affects the query.
The output format defaults to ECL - which is regenerated from the expression tree. (This ECL cannot generally be compiled without editing - partly because it contains extra annoations.) Use either of the following:
There is a debug option (-ftraceIR) that generates an intermediate representation of the expression graph rather than regenerating ECL. The output tends to be less compact and harder to read quickly, but has the advantage of being better structured, and contains more details of the internal representation. ecl/hql/hqlir.cpp contains more details of the format.
Adding extra logging into the source code
If you want to add tracing of expressions at any point in the code generation then adding either of the following calls will include the expression details in the log file:
dbglogExpr(expr); // regenerate the ecl for an expression. See other functions in ecl/hql/hqlthql.hpp
EclIR::dbglogIR(expr); // regenerate the IR for an expression. See other functions in ecl/hql/hqlir.hpp
Logging while debugging
If you are debugging inside gdb it is often useful to be able to dump out details of an expression. Calling EclIR:dump_ir(expr); will generate the IR to stdout.
p EclIR::dump_ir(expr)
The function can also be used with multiple parameters. Each expression will be dumped out, but common child nodes will only be generated once. This can be very useful when trying to determine the difference between two expressions. The quickest way is to call EclIR::dump_ir(expr1, expr2). The first difference between the expressions will be the expression that follows the first "return".
Expression sequence ids.
Sometimes it can be hard to determine where a particular IHqlExpression node was created. If that is the case, then defining DEBUG_TRACK_INSTANCEID (in ecl/hql/hqlexpr.ipp) will add a unique sequence number to each IHqlExpression that is created. There is also a function checkSeqId() at the start of ecl/hql/hqlexpr.cpp which is called whenever an expression is created, linked, released etc.. Setting a breakpoint in that function can allow you to trace back exactly when and why a particular node was created.
The key data structure within eclcc is the graph representation. The design has some key elements.
Once a node is created it is never modified.
Some derived information (e.g., sort order, number of records, unique hash, ...) might be calculated and stored in the class after it has been created, but that doesn't change what the node represents in any way. Some nodes are created in stages - e.g., records, modules. These nodes are marked as fully completed when closeExpr() is called, after which they cannot be modified.
Nodes are always commoned up.
If the same operator has the same arguments and type then there will be a unique IHqlExpression to represent it. This helps ensure that graphs stay as graphs and don't get converted to trees. It also helps with optimizations, and allows code duplicated in two different contexts to be brought together.
The nodes are link counted.
Link counts are used to control the lifetime of the expression objects. Whenever a reference to an expression node is held, its link count is increased, and decreased when no longer required. The node is freed when there are no more references. (This generally works well, but does give us problems with forward references.)
The access to the graph is through interfaces.
The main interfaces are IHqlExpression, IHqlDataset and IHqlScope. They are all defined in hqlexpr.hpp. The aim of the interfaces is to hide the implementation of the expression nodes so they can be restructured and changed without affecting any other code.
The expression classes use interfaces and a type field rather than polymorphism. This could be argued to be bad object design...but.
There are more than 500 different possible operators. If a class was created for each of them the system would quickly become unwieldy. Instead there are several different classes which model the different types of expression (dataset/expression/scope).
The interfaces contain everything needed to create and interrogate an expression tree, but they do not contain functionality for directly processing the graph.
To avoid some of the shortcomings of type fields there are various mechanisms for accessing derived attributes which avoid interrogating the type field.
Memory consumption is critical.
It is not unusual to have 10M or even 100M nodes in memory as a query is being processed. At that scale the memory consumption of each node matters - so great care should be taken when considering increasing the size of the objects. The node classes contain a class hierarchy which is there purely to reduce the memory consumption - not to reflect the functionality. With no memory constraints they wouldn't be there, but removing a single pointer per node can save 1Gb of memory usage for very complex queries.
This is the interface that is used to walk and interrogate the expression graph once it has been created. Some of the main functions are: getOperator() What does this node represent? It returns a member of the node_operator enumerated type. numChildren() How many arguments does node have? queryChild(unsigned n) What is the nth child? If the argument is out of range it returns NULL. queryType() The type of this node. queryBody() Used to skip annotations (see below) queryProperty() Does this node have a child which is an attribute that matches a given name. (see below for more about attributes). queryValue() For a no_constant return the value of the constant. It returns NULL otherwise.
The nodes in the expression graph are created through factory functions. Some of the expression types have specialised functions - e.g., createDataset, createRow, createDictionary, but scalar expressions and actions are normally created with createValue().
Note: Generally ownership of the arguments to the createX() functions are assumed to be taken over by the newly created node.
The values of the enumeration constants in node_operator are used to calculate "crcs" which are used to check if the ECL for a query matches, and if disk and index record formats match. It contains quite a few legacy entries no_unusedXXX which can be used for new operators (otherwise new operators must be added to the end).
This interface is implemented by records, and is used to map names to the fields within the records. If a record contains IFBLOCKs then each of the fields in the ifblock is defined in the IHqlSimpleScope for the containing record.
Normally obtained by calling IHqlExpression::queryScope(). It is primarily used in the parser to resolve fields from within modules.
The ECL is parsed on demand so as the symbol is looked up it may cause a cascade of ECL to be compiled. The lookup context (HqlLookupContext ) is passed to IHqlScope::lookupSymbol() for several reasons:
It contains information about the active repository - the source of the ECL which will be dynamically parsed.
It contains caches of expanded functions - to avoid repeating expansion transforms.
Some members are used for tracking definitions that are read to build dependency graphs, or archives of submitted queries.
The interface IHqlScope currently has some members that are used for creation; this should be refactored and placed in a different interface.
This is normally obtained by calling IHqlExpression::queryDataset(). It has shrunk in size over time, and could quite possibly be folded into IHqlExpression with little pain.
There is a distinction in the code generator between "tables" and "datasets". A table (IHqlDataset::queryTable()) is a dataset operation that defines a new output record. Any operation that has a transform or record that defines an output record (e.g., PROJECT,TABLE) is a table, whilst those that don't (e.g., a filter, dedup) are not. There are a few apparent exceptions -e.g., IF (This is controlled by definesColumnList() which returns true the operator is a table.)
There are two related by slightly different concepts. An attribute refers to the explicit flags that are added to operators (e.g., , LOCAL, KEEP(n) etc. specified in the ECL or some internal attributes added by the code generator). There are a couple of different functions for creating attributes. createExtraAttribute() should be used by default. createAttribute() is reserved for an attribute that never has any arguments, or in unusual situations where it is important that the arguments are never transformed. They are tested using queryAttribute()/hasAttribute() and represented by nodes of kind no_attr/no_expr_attr.
The term "property" refers to computed information (e.g., record counts) that can be derived from the operator, its arguments and attributes. They are covered in more detail below.
Fields can be selected from active rows of a dataset in three main ways:
Some operators define LEFT/RIGHT to represent an input or processed dataset. Fields from these active rows are referenced with LEFT.<field-name>. Here LEFT or RIGHT is the "selector".
Other operators use the input dataset as the selector. E.g., myFile(myFile.id != 0). Here the input dataset is the "selector".
Often when the input dataset is used as the selector it can be omitted. E.g., myFile(id != 0). This is implicitly expanded by the PARSER to the second form. A reference to a field is always represented in the expression graph as a node of kind no_select (with createSelectExpr). The first child is the selector, and the second is the field. Needless to say there are some complications...
LEFT/RIGHT.
The problem is that the different uses of LEFT/RIGHT need to be disambiguated since there may be several different uses of LEFT in a query. This is especially true when operations are executed in child queries. LEFT is represented by a node no_left(record, selSeq). Often the record is sufficient to disambiguate the uses, but there are situations where it isn't enough. So in addition no_left has a child which is a selSeq (selector sequence) which is added as a child attribute of the PROJECT or other operator. At parse time it is a function of the input dataset that is later normalized to a unique id to reduce the transformation work.
Active datasets. It is slightly more complicated - because the dataset used as the selector can be any upstream dataset up to the nearest table. So the following ECL code is legal:
Here the reference to x.id in the definition of z is referring to a field in the input dataset.
Because of these semantics the selector in a normalized tree is actually inputDataset->queryNormalizedSelector() rather than inputDatset. This function currently returns the table expression node (but it may change in the future see below).
In some situations ECL allows child datasets to be treated as a dataset without an explicit NORMALIZE. E.g., EXISTS(myDataset.ChildDataset);
This is primarily to enable efficient aggregates on disk files to be generated, but it adds some complications with an expression of the form dataset.childdataset.grandchild. E.g.,:
In the first example dataset.childdataset within the dataset.childdataset.grandchild is a reference to a dataset that doesn't have an active cursor and needs to be iterated), whilst in the second it refers to an active cursor.
To differentiate between the two, all references to fields within datasets/rows that don't have active selectors have an additional attribute("new") as a child of the select. So a no_select with a "new" attribute requires the dataset to be created, one without is a member of an active dataset cursor.
If you have a nested row, the new attribute is added to the selection from the dataset, rather than the selection from the nested row. The functions queryDatasetCursor() and querySelectorDataset()) are used to help interpret the meaning.
(An alternative would be to use a different node from no_select - possibly this should be considered - it would be more space efficient.)
The expression graph generated by the ECL parser doesn't contain any new attributes. These are added as one of the first stages of normalizing the expression graph. Any code that works on normalized expressions needs to take care to interpret no_selects correctly.
When an expression graph is transformed and none of the records are changed, the representation of LEFT/RIGHT remains the same. This means any no_select nodes in the expression tree will also stay the same.
However, if the transform modifies a table (highly likely) it means that the selector for the second form of field selector will also change. Unfortunately this means that transforms often cannot be short-circuited.
It could significantly reduce the extent of the graph that needs traversing, and the number of nodes replaced in a transformed graph if this could be avoided. One possibility is to use a different value for dataset->queryNormalizedSelector() using a unique id associated with the table. I think it would be a good long term change, but it would require unique ids (similar to the selSeq) to be added to all table expressions, and correctly preserved by any optimization.
Sometimes it is useful to add information into the expression graph (e.g., symbol names, position information) that doesn't change the meaning, but should be preserved. Annotations allow information to be added in this way.
An annotation's implementation of IHqlExpression generally delegates the majority of the methods through to the annotated expression. This means that most code that interrogates the expression graph can ignore their presence, which simplifies the caller significantly. However transforms need to be careful (see below).
Information about the annotation can be obtained by calling IHqlExpression:: getAnnotationKind() and IHqlExpression:: queryAnnotation().
In legacy ECL you will see code like the following::
EXPORT a(x) := FUNCTION
+ Y := F(x);
+ OUTPUT(Y);
+ RETURN G(Y);
+END;
+
The assumption is that whenever a(x) is evaluated the value of Y will be output. However that doesn't particularly fit in with a declarative expression graph. The code generator creates a special node (no_compound) with child(0) as the output action, and child(1) as the value to be evaluated (g(Y)).
If the expression ends up being included in the final query then the action will also be included (via the no_compound). At a later stage the action is migrated to a position in the graph where actions are normally evaluated.
There are many pieces of information that it is useful to know about a node in the expression graph - many of which would be expensive to recomputed each time there were required. Eclcc has several mechanisms for caching derived information so it is available efficiently.
Boolean flags - getInfoFlags()/getInfoFlags2().
There are many Boolean attributes of an expression that are useful to know - e.g., is it constant, does it have side-effects, does it reference any fields from a dataset etc. etc. The bulk of these are calculated and stored in a couple of members of the expression class. They are normally retrieved via accessor functions e.g., containsAssertKeyed(IHqlExpression*).
Active datasets - gatherTablesUsed().
It is very common to want to know which datasets an expression references. This information is calculated and cached on demand and accessed via the IHqlExpression::gatherTablesUsed() functions. There are a couple of other functions IHqlExpression::isIndependentOfScope() and IHqlExpression::usesSelector() which provide efficient functions for common uses.
Information stored in the type.
Currently datasets contain information about sort order, distribution and grouping as part of the expression type. This information should be accessed through the accessor functions applied to the expression (e.g., isGrouped(expr)). At some point in the future it is planned to move this information as a general derived property (see next).
Other derived property.
There is a mechanism (in hqlattr) for calculating and caching an arbitrary derived property of an expression. It is currently used for number of rows, location-independent representation, maximum record size etc. . There are typically accessor functions to access the cached information (rather than calling the underlying IHqlExpression::queryAttribute() function).
Helper functions.
Some information doesn't need to be cached because it isn't expensive to calculate, but rather than duplicating the code, a helper function is provided. E.g., queryOriginalRecord() and hasUnknownTransform(). They are not part of the interface because the number would make the interface unwieldy and they can be completely calculated from the public functions.
However, it can be very hard to find the function you are looking for, and they would greatly benefit from being grouped e.g., into namespaces.
One of the key processes in eclcc is walking and transforming the expression graphs. Both of these are covered by the term transformations. One of the key things to bear in mind is that you need to walk the expression graph as a graph, not as a tree. If you have already examined a node once you shouldn't repeat the work - otherwise the execution time may be exponential with node depth.
Other things to bear in mind
If a node isn't modified don't create a new one - return a link to the old one.
You generally need to walk the graph and gather some information before creating a modified graph. Sometimes creating a new graph can be short-circuited if no changes will be required.
Sometimes you can be tempted to try and short-circuit transforming part of a graph (e.g., the arguments to a dataset activity), but because of the way references to fields within dataset work that often doesn't work.
If an expression is moved to another place in the graph, you need to be very careful to check if the original context was conditional and that the new context is not.
The meaning of expressions can be context dependent. E.g., References to active datasets can be ambiguous.
Never walk the expressions as a tree, always as a graph!
Be careful with annotations.
It is essential that an expression that is used in different contexts with different annotations (e.g., two different named symbols) is consistently transformed. Otherwise it is possible for a graph to be converted into a tree. E.g.,:
A := x; B := x; C = A + B;
+
must not be converted to:
A' := x'; B' := X''; C' := A' + B';
+
For this reason most transformers will check if expr->queryBody() matches expr, and if not will transform the body (the unannotated expression), and then clone any annotations.
Some examples of the work done by transformations are:
Constant folding.
Expanding function calls.
Walking the graph and reporting warnings.
Optimizing the order and removing redundant activities.
Reducing the fields flowing through the generated graph.
Spotting common sub expressions.
Calculating the best location to evaluate an expression (e.g., globally instead of in a child query).
Many, many others.
Some more details on the individual transforms are given below..
The first job of eclcc is to parse the ECL into an expression graph. The source for the ECL can come from various different sources (archive, source files, remote repository). The details are hidden behind the IEclSource/IEclSourceCollection interfaces. The createRepository() function is then used to resolve and parse the various source files on demand.
Several things occur while the ECL is being parsed:
Function definitions are expanded inline.
A slightly unusual behaviour. It means that the expression tree is a fully expanded expression -which is better suited to processing and optimizing.
Some limited constant folding occurs.
When a function is expanded, often it means that some of the test conditions are always true/false. To reduce the transformations the condition may be folded early on.
When a symbol is referenced from another module this will recursively cause the ECL for that module (or definition within that module) to be parsed.
Currently the semantic checking is done as the ECL is parsed.
If we are going to fully support template functions and delayed expansion of functions this will probably have to change so that a syntax tree is built first, and then the semantic checking is done later.
There are various problems with the expression graph that comes out of the parser:
Records can have values as children (e.g., { myField := infield.value} ), but it causes chaos if record definitions can change while other transformations are going on. So the normalization removes values from fields.
Some activities use records to define the values that output records should contain (e.g., TABLE). These are now converted to another form (e.g., no_newusertable).
Sometimes expressions have multiple definition names. Symbols and annotations are rationalized and commoned up to aid commoning up other expressions.
Some PATTERN definitions are recursive by name. They are resolved to a form that works if all symbols are removed.
The CASE/MAP representation for a dataset/action is awkward for the transforms to process. They are converted to nested Ifs.
(At some point a different representation might be a good idea.)
EVALUATE is a weird syntax. Instances are replaced with equivalent code which is much easier to subsequently process.
The datasets used in index definitions are primarily there to provide details of the fields. The dataset itself may be very complex and may not actually be used. The dataset input to an index is replaced with a dummy "null" dataset to avoid unnecessary graph transforming, and avoid introducing any additional incorrect dependencies.
Generally if you use LEFT/RIGHT then the input rows are going to be available wherever they are used. However if they are passed into a function, and that function uses them inside a definition marked as global then that is invalid (since by definition global expressions don't have any context).
Similarly if you use syntax <dataset>.<field>, its validity and meaning depends on whether <dataset> is active. The scope transformer ensures that all references to fields are legal, and adds a "new" attribute to any no_selects where it is necessary.
This transform simplifies the expression tree. Its aim is to simplify scalar expressions, and dataset expressions that are valid whether or not the nodes are shared. Some examples are:
1 + 2 => 3 and any other operation on scalar constants.
IF(true, x, y) => x
COUNT(<empty-dataset>) => 0
IF (a = b, 'c', 'd') = 'd' => IF(a=b, false, true) => a != b
Simplifying sorts, projects filters on empty datasets
Most of the optimizations are fairly standard, but a few have been added to cover more esoteric examples which have occurred in queries over the years.
This transform also supports the option to percolate constants through the graph. E.g., if a project assigns the value 3 to a field, it can substitute the value 3 wherever that field is used in subsequent activities. This can often lead to further opportunities for constant folding (and removing fields in the implicit project).
This transformer is used to simplify, combine and reorder dataset expressions. The transformer takes care to count the number of times each expression is used to ensure that none of the transformations cause duplication. E.g., swapping a filter with a sort is a good idea, but if there are two filters of the same sort and they are both swapped you will now be duplicating the sort.
ECL tends to be written as general purpose definitions which can then be combined. This can lead to potential inefficiencies - e.g., one definition may summarise some data in 20 different ways, this is then used by another definition which only uses a subset of those results. The implicit project transformer tracks the data flow at each point through the expression graph, and removes any fields that are not required.
This often works in combination with the other optimizations. For instance the constant percolation can remove the need for fields, and removing fields can sometimes allow a left outer join to be converted to a project.
is this the correct term? Should it be a query? This should really be independent of this document...)
The code generator ultimately creates workunits. A workunit completely describes a generated query. It consists of two parts. There is an xml component - this contains the workflow information, the various execution graphs, and information about options. It also describes which inputs can be supplied to the query and what results are generated. The other part is the generated shared object compiled from the generated C++. This contains functions and classes that are used by the engines to execute the queries. Often the xml is compressed and stored as a resource within the shared object -so the shared object contains a complete workunit.
The actions in a workunit are divided up into individual workflow items. Details of when each workflow item is executed, what its dependencies are stored in the <Workflow> section of the xml. The generated code also contains a class definition, with a method perform() which is used to execute the actions associated with a particular workflow item. (The class instances are created by calling the exported createProcess() factory function).
The generated code for an individual workflow item will typically call back into the engine at some point to execute a graph.
The activity graphs are stored in the xml. The graph contains details of which activities are required, how those activities link together, what dependencies there are between the activities. For each activity it might contain the following information:
A unique id.
The "kind" of the activity (from enum ThorActivityKind in eclhelper.hpp)
The ECL that created the activity.
Name of the original definition
Location (e.g., file, line number) of the original ECL.
Information about the record size, number of rows, sort order etc.
Hints which control options for a particular activity (e.g,, the number of threads to use while sorting).
Record counts and stats once the job has executed.
Each activity in a graph also has a corresponding helper class instance in the generated code. (The name of the class is cAc followed by the activity number, and the exported factory method is fAc followed by the activity number.) These classes implement the interfaces defined in eclhelper.hpp.
The engine uses the information from the xml to produce a graph of activities that need to be executed. It has a general purpose implementation of each activity kind, and it uses the class instance to tailor that general activity to the specific use e.g., what is the filter condition, what fields are set up, what is the sort order?
The workunit xml contains details of what inputs can be supplied when that workunit is run. These correspond to STORED definitions in the ECL. The result xml also contains the schema for the results that the workunit will generate.
Once an instance of the workunit has been run, the values of the results may be written back into dali's copy of the workunit so they can be retrieved and displayed.
Compile time is an issue - especially for small on-demand queries. To help reduce compile times (and dependencies with the rest of the system) the number of header files included by the generated code is kept to a minimum. In particular references to jlib, boost and icu are kept within the implementation of the runtime functions, and are not included in the public dependencies.
Thread-safe.
It should be possible to use the members of an activity helper from multiple threads without issue. The helpers may contain some context dependent state, so different instances of the helpers are needed for concurrent use from different contexts (e.g., expansions of a graph.)
Concise.
The code should be broadly readable, but the variable names etc. are chosen to generate compact code.
Functional.
Generally the generated code assigns to a variable once, and doesn't modify it afterwards. Some assignments may be conditional, but once the variable is evaluated it isn't updated. (There are of course a few exceptions - e.g., dataset iterators)
First a few pointers to help understand the code within eclcc:
It makes extensive use of link counting. You need understand that concept to get very far.
If something is done more than once then that is generally split into a helper function.
The helper functions aren't generally added to the corresponding interface (e.g., IHqlExpression) because the interface would become bloated. Instead they are added as global functions. The big disadvantage of this approach is they can be hard to find. Even better would be for them to be rationalised and organised into namespaces.
The code is generally thread-safe unless there would be a significant performance implication. In generally all the code used by the parser for creating expressions is thread safe. Expression graph transforms are thread-safe, and can execute in parallel if a constant (NUM_PARALLEL_TRANSFORMS) is increased. The data structures used to represent the generated code are NOT thread-safe.
Much of the code generation is structured fairly procedurally, with classes used to process the stages within it.
There is a giant "God" class HqlCppTranslator - which could really do with refactoring.
The eclcc parser uses the standard tools bison and flex to process the ECL and convert it to a
: expression graph. There are a couple of idiosyncrasies with the way it is implemented.
Macros with fully qualified scope.
Slightly unusually macros are defined in the same way that other definitions are - in particular to can have references to macros in other modules. This means that there are references to macros within the grammar file (instead of being purely handled by a pre-processor). It also means the lexer keeps an active stack of macros being processed.
Attributes on operators.
Many of the operators have optional attributes (e.g., KEEP, INNER, LOCAL, ...). If these were all reserved words it would remove a significant number of keywords from use as symbols, and could also mean that when a new attribute was added it broke existing code. To avoid this the lexer looks ahead in the parser tables (by following the potential reductions) to see if the token really could come next. If it can't then it isn't reserved as a symbol.
As the workunit is created the code generator builds up the generated code and the xml for the workunit. Most of the xml generation is encapsulated within the IWorkUnit interface. The xml for the graphs is created in an IPropertyTree, and added to the workunit as a block.
The C++ generation is ultimately controlled by some template files (thortpl.cpp). The templates are plain text and contain references to allow named sections of code to be expanded at particular points.
The code generator builds up some structures in memory for each of those named sections. Once the generation is complete some peephole optimization is applied to the code. This structure is walked to expand each named section of code as required.
The BuildCtx class provides a cursor into that generated C++. It will either be created for a given named section, or more typically from another BuildCtx. It has methods for adding the different types of statements. Some are simple (e.g., addExpr()), whilst some create a compound statement (e.g., addFilter). The compound statements change the active selector so any new statements are added within that compound statement.
As well as building up a tree of expressions, this data structure also maintains a tree of associations. For instance when a value is evaluated and assigned to a temporary variable, the logical value is associated with that temporary. If the same expression is required later, the association is matched, and the temporary value is used instead of recalculating it. The associations are also used to track the active datasets, classes generated for row-meta information, activity classes etc. etc.
Each activity in an expression graph will have an associated class generated in the C++. Each different activity kind expects a helper that implements a particular IHThorArg interface. E.g., a sort activity of kind TAKsort requires a helper that implements IHThorSortArg. The associated factory function is used to create instances of the helper class.
The generated class might take one of two forms:
A parameterised version of a library class. These are generated for simple helpers that don't have many variations (e.g., CLibrarySplitArg for TAKsplit), or for special cases that occur very frequently (CLibraryWorkUnitReadArg for internal results).
A class derived from a skeleton implementation of that helper (typically CThorXYZ implementing interface IHThorXYZ). The base class has default implementations of some of the functions, and any exceptions are implemented in the derived class.
This is a class that is used by the engines to encapsulate all the information about a single row -e.g., the format that each activity generates. It is an implementation of the IOutputMeta interface. It includes functions to
The same expression nodes are used for representing expressions in the generated C++ as the original ECL expression graph. It is important to keep track of whether an expression represents untranslated ECL, or the "translated" C++. For instance ECL has 1 based indexes, while C++ is zero based. If you processed the expression x[1] it might get translated to x[0] in C++. Translating it again would incorrectly refer to x[-1].
There are two key classes used while building the C++ for an ECL expression:
CHqlBoundExpr.
This represents a value that has been converted to C++. Depending on the type, one or more of the fields will be filled in.
CHqlBoundTarget.
This represents the target of an assignment -C++ variable(s) that are going to be assigned the result of evaluating an expression. It is almost always passed as a const parameter to a function because the target is well-defined and the function needs to update that target.
A C++ expression is sometimes converted back to an ECL pseudo-expression by calling getTranslatedExpr(). This creates an expression node of kind no_translated to indicate the child expression has already been converted.
The generation code for expressions has a hierarchy of calls. Each function is there to allow optimal code to be generated - e.g., not creating a temporary variable if none are required. A typical flow might be:
buildExpr(ctx, expr, bound).
Evaluate the ecl expression "expr" and save the C++ representation in the class bound. This might then call through to...
buildTempExpr(ctx, expr, bound);
Create a temporary variable, and evaluate expr and assign it to that temporary variable.... Which then calls.
buildExprAssign(ctx, target, expr);
evaluate the expression, and ensure it is assigned to the C++ target "target".
The default implementation might be to call buildExpr....
An operator must either be implemented in buildExpr() (calling a function called doBuildExprXXX) or in buildExprAssign() (calling a function called doBuildAssignXXX). Some operators are implemented in both places if there are different implementations that would be more efficient in each context.
Similarly there are several different assignment functions:
buildAssign(ctx, <ecl-target>, <ecl-value>);
buildExprAssign(ctx, <c++-target>, <ecl-value>);
assign(ctx, <C++target>, <c++source>)
The different varieties are there depending on whether the source value or targets have already been translated. (The names could be rationalised!)
Most dataset operations are only implemented as activities (e.g., PARSE, DEDUP). If these are used within a transform/filter then eclcc will generate a call to a child query. An activity helper for the appropriate operation will then be generated.
However a subset of the dataset operations can also be evaluated inline without calling a child query. Some examples are filters, projects, and simple aggregation. It removes the overhead of the child query call in the simple cases, and often generates more concise code.
When datasets are evaluated inline there is a similar hierarchy of function calls:
buildDatasetAssign(ctx, target, expr);
Evaluate the dataset expression, and assign it to the target (a builder interface). This may then call....
buildIterate(ctx, expr)
Iterate through each of the rows in the dataset expression in turn. Which may then call...
buildDataset(ctx, expr, target, format)
Build the entire dataset, and return it as a single value.
Some of the operations (e.g., aggregating a filtered dataset) can be done more efficiently by summing and filtering an iterator, than forcing the filtered dataset to be evaluated first.
The interface IHqlCppDatasetCursor allows the code generator to iterate through a dataset, or select a particular element from a dataset. It is used to hide the different representation of datasets, e.g.,
Blocked - the rows are in a contiguous block of memory appended one after another.
Array - the dataset is represented by an array of pointers to the individual rows.
Link counted - similar to array, but each element is also link counted.
Nested. Sometimes the cursor may iterate through multiple levels of child datasets.
Generally rows that are serialized (e.g., on disk) are in blocked format, and they are stored as link counted rows in memory.
The IReferenceSelector interface and the classes in hqltcppc[2] provide an interface for getting and setting values within a row of a dataset. They hide the details of the layout - e.g., csv/xml/raw data, and the details of exactly how each type is represented in the row.
The current implementation of keys in HPCC uses a format which uses a separate 8 byte integer field which was historically used to store the file position in the original file. Other complications are that the integer fields are stored big-endian, and signed integer values are biased.
This introduces some complication in the way indexes are handled. You will often find that the logical index definition is replaced with a physical index definition, followed by a project to convert it to the logical view. A similar process occurs for disk files to support VIRTUAL(FILEPOSITION) etc.
As mentioned at the start of this document, one of the main challenges with eclcc is converting the declarative ECL code into imperative C++ code. The direction we are heading in is to allow the engines to support more lazy-evaluation so possibly in this instance to evaluate it the first time it is used (although that may potentially be much less efficient). This will allow the code generator to relax some of its current assumptions.
There are several example queries which are already producing pathological behaviour from eclcc, causing it to generate C++ functions which are many thousands of lines long.
Currently the grammar for the parser is too specialised. In particular the separate productions for expression, datasets, actions cause problems - e.g., it is impossible to properly allow sets of datasets to be treated in the same way as other sets.
The semantic checking (and probably semantic interpretation) is done too early. Really the parser should build up a syntax tree, and then disambiguate it and perform the semantic checks on the syntax tree.
The function calls should probably be expanded later than they are. I have tried in the past and hit problems, but I can't remember all the details. Some are related to the semantic checking.
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/CodeReviews.html b/devdoc/CodeReviews.html
new file mode 100644
index 00000000000..859d5706f12
--- /dev/null
+++ b/devdoc/CodeReviews.html
@@ -0,0 +1,24 @@
+
+
+
+
+
+ Code Review Guidelines | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
The Code Submissions document is aimed at developers that are submitting PRs. This document describes some of the goals and expectations for code reviewers.
Catch architectural or design problems. These should have been caught earlier, but better later than never...
Catch bugs early (incorrect behaviour, inefficiencies, security issues)
Ensure the code is readable and maintainable. This includes following the project coding standards (see Style Guide).
A opportunity for training/passing on information. For example providing information about how the current system works, functionality that is already available or suggestions of other approaches the developer may not have thought of.
It is NOT a goal to change the submission until it matches how the reviewer would have coded it.
Code reviewers should be explicit and clearly describe the problem. This should include what change is expected if not obvious. Don’t assume the contributor has same understanding/view as reviewer.
If a comment is not clear the contributor should ask for clarification. ...rather than wasting time trying to second-guess the reviewer.
Contributors should feel free to push back if they consider comments are too picky. The reviewer can either agree, or provide reasons why they consider it to be an issue.
The reviewer should not extend the scope of the original change. If the change could be extended, or only partially solves the issue, a new JIRA should be created for the extra work. If the change will introduce regressions, or fundamentally fails to solve the problem then this does not apply!
Clearly indicate if a review is incomplete. Sometimes a significant design problem means the rest of the code has not been reviewed in detail. Other times an initial review has picked up a set of issues, but the reviewer needs to go back and check other aspects in detail. If this is the case it should be explicitly noted.
Repeated issues. The reviewer is free to comment on every instance of a repeated issue, but a simple annotation should alert contributor to address appropriately eg: [Please address all instances of this issue]
Contributers should provide feedback to the reviewer. The contributor should respond to a comment if it isn't obvious where/how they have been addressed (but no need to acknowledge typo/indentation/etc)
Only the reviewer should mark issues as resolved using the Github resolve conversation button.
Code reviews should be a priority. Both reviewers and contributors should respond in a timely manner - don't leave it for days. It destroys the flow of thought and conversation.
Check all review comments have been addressed. If they have not been addressed you are guaranteed another review/submit cycle. In particular watch out for collapsed conversations. If there are large numbers of comments GitHub will collapse them, which can make comments easy to miss.
Sometimes PRs need to be restarted. If there are large number of comments > 100, it can be hard to track all the comments and GitHub can become unresponsive. It may be better to close the PR and open a new one.
Submit any changes as extra commits. This makes it clear to the reviewer what has changed, and avoids them having to re-review everything. Please do not squash them until the reviewers approve the PR. The few exceptions to this are if the PR is only a couple of lines, or the PR is completely rewritten in response to the review.
Reviewers use GitHub's features Making use of the "viewed" button can make it easier to track what has changed - or quickly remove trivial changes from view. Ignoring whitespace can often simplify comparisons - especially when code has been refactored or extra conditions or try/catch bocks have been introduced.
All code reviews don't need to be equally strict. The "strictness" of the review should reflect the importance and location of the change. Some examples:
If it is closely associated with an existing file, then the indentation, style should match the surrounding code - a mixture of styles makes it much harder to read. If it is in a new, independent source file or project this is less of an issue.
If the code is in a core library then efficiency and edge cases will be more important.
If it is a core part of the system then security is key. If it is a developer only tool then edge cases are less significant.
Reviews of draft pull requests are likely to concentrate on the overall approach, rather than the details. They are likely to be more informal (e.g. not always using comments tags).
Does it introduce any memory leaks? E.g. Correct use of linking? Are exceptions released?
Thread safety
critical sections or atomic variables if accessed by more than one thread
race conditions
deadlock
authorization. Should it be checked, does it fail by default?
Any potential for overflow or DOS? Are all user inputs validated and all lengths protected?
Are all secrets stored and passed securely?
Comments explaining why for any code that is complex or counter-intuitive.
Backward compatibility. Could this possibly cause problems if data produced with this change is used in earlier/later versions? Could there be problems if it was used in a mixed-version environment?
When reading comments in a review it can sometimes be hard to know why the reviewer made a comment, or what response is expected. If there is any doubt the contributor should ask. However to make it clearer we are aiming to always add a tag to the front of each review comment. The tag will give an indication of why the comment is being made, its severity and what kind of response is expected. Here is a provisional table of tags:
Tag
What
Why
Expected response
design:
An architectural or design issue
The reviewer considers the PR has a significant problem which will affect its functionality or future extensibility
reviewer/developer redesign expected before any further changes
scope:
The scope of the PR does not match the Jira
If the scope of the fix is too large it can be hard to review, and take much longer to resolve all the issues before the PR is accepted.
Discussion. Split the PR into multiple simpler PRs.
function:
Incorrect/unexpected functionality implemented
The function doesn't match the description in the jira, or doesn't solve the original problem
developer expected to address issue (or discuss)
security:
Something in the code introduces a security problem
The reviewer has spotted potential security issues e.g. injection attacks
developer expected to discuss the issue (and then address)
bug:
A coding issue that will cause incorrect behaviour
Likely to cause confusion, invalid results or crashes.
developer expected to address issue
efficiency:
The code works, but may have scaling or other efficiency issues.
Inefficiency can cause problem in some key functions and areas
developer addressing the problem (or discuss)
discuss:
Reviewer has thought of a potential problem, but not sure if it applies
Reviewer has a concern it may be an issue, and wants to check the developer has thought about and addressed the issue
Discussion - either in the PR or offline.
style:
Reviewer points out non-conforming code style
Makes the code hard to read
Developer to fix
indent:
A fairly obvious indentation issue
Makes the code hard to read
Developer to fix.
format:
Any other unusual formatting
Makes the code hard to read
Developer to fix.
typo:
Minor typing error
Makes something (code/message/comment) harder to read
Developer to fix.
minor:
A minor issue that could be improved.
Education (the suggestion is better for a particular reason), or something simple to clean up at the same time as other changes
Developer recommended to fix, but unlikely to stop a merge
picky:
A very minor issue that could be improved, but is barely worth commenting on
Education, or something to clean up at the same time as other changes
Developer discretion to fix, wouldn't stop a merge
future:
An additional feature or functionality that fits in but should be done as a separate PR.
Ensure that missing functionality is tracked, but PRs are not held up by additional requirements.
Contributor to create Jira (unless trivial) and number noted in response.
question:
Review has a question that they are not sure of the answer to
Reviewer would like clarification to help understand the code or design. The answer may lead to further comments.
An answer to the question.
note:
Reviewer wants to pass on some information to the contributor which they may not know
Passing on knowledge/background
contributor should consider the note, but no change expected/required
personal:
Reviewer has an observation based on personal experience
Reviewer has comments that would improve the code, but not part of the style guide or required. E.g. patterns for guard conditions
Reflect on the suggestion, but no change expected.
documentation:
This change may have an impact on documentation
Make sure changes can be used
Contributor to create Jira describing the impact created, and number noted in response.
The comments should always be constructive. The reviewer should have a reason for each of them, and be able to articulate the reason in the comment or when asked. "I wouldn't have done it like that" is not a good enough on its own!
Similarly there is a difference in opinion within the team on some style issues - e.g. standard libraries or jlib, inline or out of line functions, nested or non-nested classes. Reviews should try and avoid commenting on these unless there is a clear reason why they are significant (functionality, efficiency, compile time) and if so spell it out. Code reviewers should discuss any style issues that they consider should be universally adopted that are not in the style guide.
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/CodeSubmissions.html b/devdoc/CodeSubmissions.html
new file mode 100644
index 00000000000..373fcd1cfbe
--- /dev/null
+++ b/devdoc/CodeSubmissions.html
@@ -0,0 +1,24 @@
+
+
+
+
+
+ Code Submission Guidelines | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
We welcome submissions to the platform especially in the form of pull requests into the HPCC-Systems github repository. The following describes some of processes for merging PRs.
There are a few things that should be considered when creating a PR to increase the likelihood that they can be accepted quickly.
Write a good commit message The format should be HPCC-XXXXX (where XXXXX is the bug number) followed by a description of the issue. The text should make sense in a change log by itself - without reference to the jira or the contents of the PR. We should aim to increase the information that is included as part of the commit message - not rely on on the jira.
Ensure the reviewer has enough information to review the change. The code reviewer only has the JIRA and the PR to go on. The JIRA (or associated documentation) should contain enough details to review the PR - e.g. the purpose, main aim, why the change was made etc.. If the scope of the jira has changed then the jira should be updated to reflect that. If the submission requires changes to the documentation then the JIRA should contain all the details needed to document it, and the PR should either contain the documentation changes, or a documentation JIRA should be created.
Fill in the checklist The check boxes are there to remind you to consider different aspects of the PR. Not all of them apply to every submission, but if you tick a box and have not really thought about the item then prepare to be embarrassed!
Prefer small submissions It isn't always possible, but several smaller PRs are much easier to review than one large change. If your submission includes semi-automatic/mechanical changes (e.g. renaming large numbers of function calls, or adding an extra parameter) please keep it as a separate commit. This makes it much easier to review the PR - since the reviewer will be looking for different errors in the different types of changes.
Check for silly mistakes Review your own code in github, after creating the PR to check for silly mistakes. It doesn't take long, and often catches trivial issues. It may avoid the need for a cycle of code-review/fixes. It may be helpful to add some notes to specific changes e.g. "this change is mainly or solely refactoring method A into method B and C. ". Some common examples of trivial issues to look for include:
Inconsistent indentation, or using tabs rather than spaces to indent.
Lines of tracing left in.
Lines of code commented out that should be deleted.
TBD reminders of work that need implementing or removing.
Unrelated files that have been accidentally modified.
Accidental changes to submodule versions.
Typos in error messages, tracing or comments, or in the commit message.
Incomplete edits when copy and pasting code.
Check subtractions are the right way around, and potential for overflow.
New files with the wrong copyright date
Check the target branch (see below)
Request one or more reviews. For relatively simple changes a single reviewer is normally enough.
All pull requests should be reviewed by someone who is not the author before merging. Complex changes, changes that require input from multiple experts, or that have implications throughout the system should be reviewed by multiple reviewers. This should include someone who is responsible for merging changes for that part of the system. (Unless it is a simple change written by someone with review rights.)
Contributors should use the github reviewers section on the PR to request reviews. After a contributor has pushed a set of changes in response to a review, they should refresh the github review status, so the users are notified it is ready for re-review. When the review is complete, a person with responsibility for merging changes to that part of the system should be added as a reviewer (or refreshed), with a comment that it is ready to merge.
Reviewers should check for PRs that are ready for their review via github's webpage (filter "review-requested:<reviewer-id>") or via the github CLI (e.g. gh pr status). Contributors should similarly ensure they stay up to date with any comments on requests for change on their submissions.
The Version support document contains details of the different versions that are supported, and which version should be targetted for different kinds of changes. Occasionally earlier branches will be chosen, (e.g. security fixes to even older versions) but they should always be carefully discussed (and documented).
Changes will always be upmerged into the next point release for all the more recent major and minor versions (and master).
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/DevDocs.html b/devdoc/DevDocs.html
new file mode 100644
index 00000000000..b1d2b0f764c
--- /dev/null
+++ b/devdoc/DevDocs.html
@@ -0,0 +1,31 @@
+
+
+
+
+
+ Working with developer documentation | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Documents can be located anywhere in the repository folder structure. If it makes sense to have documentation "close" to specific components, then it can be located in the same folder as the component. For example, any developer documentation for specific plugins can be located in those folders. If this isn't appropriate then the documentation can be located in the devdoc or subfolders of devdoc.
WARNING
There is an exclusion list in the devdoc/.vitepress/config.js file that prevents certain folders from being included in the documentation. If you add a new document to a folder that is excluded, then it will not be included in the documentation. If you need to add a new document to an excluded folder, then you will need to update the exclusion list in the devdoc/.vitepress/config.js file.
Documentation is written in Markdown. This is a simple format that is easy to read and write. It is also easy to convert to other formats, such as HTML, PDF, and Word. Markdown is supported by many editors, including Visual Studio Code, and is supported by VitePress.
TIP
VitePress extends Markdown with some additional features, such as custom containers, it is recommended that you refer to the VitePress documentation for more details.
To assist with the writing of documentation, VitePress can be used to render the documentation locally. This allows you to see how the documentation will look when it is published. To start the local development server you need to type the following commands in the root HPCC-Platform folder:
sh
npm install
+npm run docs-dev
This will start a local development server and display the URL that you can use to view the documentation. The default URL is http://localhost:5173/HPCC-Platform, but it may be different on your machine. The server will automatically reload when you make changes to the documentation.
WARNING
The first time you start the VitePress server it will take a while to complete. This is because it is locating all the markdown files in the repository and creating the html pages. Once it has completed this step once, it will be much faster to start the server again.
To add a new document, you need to add a new markdown file to the repository. The file should be named appropriately and have the .md file extension. Once the file exists, you can view it by navigating to the appropriate URL. For example, if you add a new file called MyNewDocument.md to the devdoc folder, then you can view it by navigating to http://localhost:5173/HPCC-Platform/devdoc/MyNewDocument.html.
To add a new document to the sidebar, you need to add an entry to the devdoc/.vitepress/config.js file. The entry should be added to the sidebar section. For example, to add a new document called MyNewDocument.md to the devdoc folder, you would add the following entry to the sidebar section:
The HPCC Platform sources are hosted on GitHub. You can download a snapshot of any branch using the download button there, or you can set up a git clone of the repository. If you are planning to contribute changes to the system, see the CONTRIBUTORS document for information about how to set up a GitHub fork of the project through which pull-requests can be made.
The HPCC platform requires a number of third party tools and libraries in order to build. The HPCC Wiki contains the details of the dependencies that are required for different distributions.
For building any documentation, the following are also required:
The HPCC system is built using the cross-platform build tool cmake, which is available for Windows, virtually all flavors of Linux, FreeBSD, and other platforms. You should install cmake version 2.8.3 or later before building the sources.
On some distros you will need to build cmake from sources if the version of cmake in the standard repositories for that distro is not modern enough. It is good practice in cmake to separate the build directory where objects and executables are made from the source directory, and the HPCC cmake scripts will enforce this.
To build the sources, create a directory where the built files should be located, and from that directory, run:
bash
cmake <source directory>
Depending on your operating system and the compilers installed on it, this will create a makefile, Visual Studio .sln file, or other build script for building the system. If cmake was configured to create a makefile, then you can build simply by typing:
bash
make
If a Visual Studio solution file was created, you can load it simply by typing the name: hpccsystems-platform.sln
This will load the solution in Visual Studio where you can build in the usual way.
To make an installation package on a supported linux system, use the command:
bash
make package
This will first do a make to ensure everything is up to date, then will create the appropriate package for your operating system, Currently supported package formats are rpm (for RedHat/Centos) and .deb (for Debian and Ubuntu). If the operating system is not one of the above, or is not recognized, make package will create a tarball.
The package installation does not start the service on the machine, so if you want to give it a go or test it (see below), make sure to start the service manually and wait until all services are up (mainly wait for EclWatch to come up on port 8010).
Some components have their own unit-tests. Once you have compiled (no need to start the services), you can already run them. Supposing you build a Debug version, from the build directory you can run:
The ECLCC compiler tests rely on two distinct runs: a known good one and your test build. For normal development, you can safely assume that the OSS/master branch in github is good. For overnight testing, golden directories need to be maintained according to the test infrastructure. There are Bash (Linux) and Batch (Windows) scripts to run the regressions:
The basic idea behind this tests is to compare the output files (logs and XML files) between runs. The log files should change slightly (the comparison should be good enough to filter most irrelevant differences), but the XML files should be identical if nothing has changed. You should only see differences in the XML where you have changed in the code, or new tests were added as part of your development.
On Linux, there are two steps:
Step 1: Check-out OSS/master, compile and run the regressions to populate the 'golden' directory:
bash
./regress.sh -t golden -e buildDir/Debug/bin/eclcc
This will run the regressions in parallel, using as many CPUs as you have, and using your just-compiled ECLCC, assuming you compiled for Debug version.
Step 2: Make your changes (or check-out your branch), compile and run again, this time output to a new directory and compare to the 'golden' repo.:
bash
./regress.sh -t my_branch -c golden -e buildDir/Debug/bin/eclcc
This will run the regressions in the same way, output to 'my_branch' dir and compare it to the golden version, highlighting the differences.
NOTE: If you changed the headers that the compiled binaries will use, you must re-install the package (or provide -i option to the script to the new headers).
Step 3: Step 2 only listed the differences, now you need to see what they are. For that, re-run the regressing script omitting the compiler, since the only thing we'll do is to compare verbosely.:
bash
./regress.sh -t my_branch -c golden
This will show you all differences, using the same ignore filters as before, between your two branches. Once you're happy with the differences, commit and issue a pull-request.
On linux systems, the makefile generated by cmake will build a specific version (debug or release) of the system depending on the options selected when cmake is first run in that directory. The default is to build a release system. In order to build a debug system instead, use command:
bash
cmake -DCMAKE_BUILD_TYPE=Debug <source directory>
You can then run make or make package in the usual way to build the system.
On a Windows system, cmake always generates s solution file with both debug and release target platforms in it, so you can select which one to build within Visual Studio.
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/GitAuthenticate.html b/devdoc/GitAuthenticate.html
new file mode 100644
index 00000000000..085a9bc6f85
--- /dev/null
+++ b/devdoc/GitAuthenticate.html
@@ -0,0 +1,35 @@
+
+
+
+
+
+ HPCC git support | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Version 8.4 of the HPCC platform allows package files to define dependencies between git repositories and also allows you to compile directly from a git repository.
E.g.
ecl run hthor --main demo.main@ghalliday/gch-ecldemo-d#version1 --server=...
There are no futher requirements if the repositories are public, but private repositories have the additional complication of supplying authentication information. Git provides various methods for providing the credentials...
This is used when the github reference is of the form ssh://github.com. The sshkey can be protected with a passcode, and there are various options to avoid having to enter the passcode each time.
It is preferrable to use the https:// protocol instead of ssh:// for links in package-lock.json files. If ssh:// is used it requires any machine that processes the dependency to have access to a registered ssh key.
github authentication
Download the GitHub command line tool (https://github.com/cli/cli). You can then use it to authenticate all git access with
All of the options above are likely to involve some user interaction - passphrases for ssh keys, web interaction with github authentication, and initial entry for cached access tokens. This is problematic for eclccserver - which cannot support user interaction, and it is preferrable not to pass credentials around.
The solution is to use a personal access token securely stored as a secret. (This would generally be associated with a special service account.) This avoids the need to pass credentials and allows the keys to be rotated.
The following describes the support in the different versions:
b) add a secret to the values.yaml file, with a key that matches the username:
secrets:
+ git:
+ ghalliday: my-git-secret
note: this cannot currently use a vault - probably need to rethink that. (Possibly extract from secret and supply as an optional environment variable to be picked up by the bash script.)
c) add a secret to Kubernetes containing the personal access token:
This document covers the main steps taken by the LDAP Security Manager during initialization. It is important to note that the LDAP Security Manager uses the LDAP protocol to access an Active Directory, AD. The AD is the store for users, groups, permissions, resources, and more. The term LDAP is generally overused to refer to both.
Each service and/or component using the LDAP security manager gets its own instance of the security manager. This includes a unique connection pool (see below). All operations described apply to each LDAP instance.
The LDAP Security Manager supports using multiple ADs. The FQDN or IP address of each AD host is read from configuration data and stored internally. The source is a comma separated list stored in the ldapAddress config value. Each entry is added to a pool of ADs.
Note that all ADs are expected to use the same credentials and have the same configuration
AD credentials consist of a username and password. The LDAP security manager users these to perform all operations on behalf of users and components in the cluster. There are three potential sources for credentials.
A Kubernetes secret. If the ldapAdminSecretKey config value is set, but ldapAdminVaultId is not (see 2) then the AD credentials are retrieved as Kubernetes secrets.
If both ldapAdminSecretKey and ldapAdminVaultId config values are present, the AD credentials are retrieved from the vault.
Hardcoded values from the systemCommonName and systemPassword config values stored in the environment.xml file.
As stated above, when multiple ADs are in use, the configuration of each must be the same. This includes credentials.
During initialization, the security manager begins incrementing through the set of defined ADs until it is able to connect and retrieve information from an AD. Once retrieved, the information is used for all ADs (see statement above about all ADs being the same). The accessed AD is marked as the current AD and no other ADs are accessed during initialization.
The retrieved information is used to verify the AD type so the security manager adjusts for variations between types. Additionally, defined DNs may be adjusted to match AD type requirements.
The manager handles connections to an AD in order to perform required operations. It is possible that values such as permissions and resources may be cached to improve performance.
The LDAP security manager maintains a pool of LDAP connections. The pool is limited in size to maxConnections from the configuration. The connection pool starts empty. As connections are created, each is added to the pool until the max allowed is reached.
The following process is used when an LDAP connection is needed.
First, the connection pool is searched for a free connection. If found and valid, the connection is returned. A connection is considered free if no one is using it and valid if the AD can be accessed. If no valid free connections are found, a new uninitialized connection is created.
For a new connection, an attempt is made to connect to each AD starting with the current. See Handling AD Hosts below for how ADs are cycled when a connection fails. For each AD, as it cycles through, connection attempts are retried with a short delay between each. If unable to connect, the AD host is marked rejected and the next is attempted. Once a new connection has been established, if the max number of connections has not been reached yet, the connection is added to the pool.
It is important to note that if the pool has reached its max size, new connections will continue to be made, but are not saved in the pool. This allows the pool to maintain a steady working state, but allow for higher demand. Connections not saved to the pool are deleted once no longer in use.
The manager keeps a list of AD hosts and the index of the current host. The current host is used for all AD operations until there is a failure. At that time the manager marks the host as "rejected" and moves to the next host using a round-robin scheme.
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/MemoryManager.html b/devdoc/MemoryManager.html
new file mode 100644
index 00000000000..9c20aaedeac
--- /dev/null
+++ b/devdoc/MemoryManager.html
@@ -0,0 +1,24 @@
+
+
+
+
+
+ The Roxie Memory Manager | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
This memory manager started life as the memory manager which was only used for the Roxie engine. It had several original design goals:
Support link counted rows. (When the last reference is released the row is freed.)
Be as fast as possible on allocate and deallocate of small rows.
Allow rows serialized from slaves to be used directly without being cloned first.
Allow the memory used by a single query, or by all queries combined, to be limited, with graceful recovery.
Isolate roxie queries from one another, so that one query can't bring down all the rest by allocating too much memory.
Guarantee all the memory used by a query is freed when the query finishes, reducing the possibility of memory leaks.
Predictable behaviour with no pathogenic cases.
(Note that efficient usage of memory does not appear on that list - the expectation when the memory manager was first designed was that Roxie queries would use minimal amounts of memory and speed was more important. Some subsequent changes e.g., Packed heaps, and configurable bucket sizes help mitigate that.)
The basic design is to reserve (but not commit) a single large block of memory in the virtual address space. This memory is subdivided into "pages". (These are not the same as the os virtual memory pages. The memory manager pages are currently defined as 1Mb in size.)
The system uses a bitmap to indicate whether each page from the global memory has been allocated. All active IRowManager instances allocate pages from the same global memory space. To help reduce fragmentation allocations for single pages are fulfilled from one end of the address space, while allocations for multiple pages are fulfilled from the other.
This provides the primary interface for allocating memory. The size of a requested allocation is rounded up to the next "bucket" size, and the allocation is then satisfied by the heap associated with that bucket size. Different engines can specify different bucket sizes - an optional list is provided to setTotalMemoryLimit. Roxie tends to use fewer buckets to help reduce the number of active heaps. Thor uses larger numbers since it is more important to minimize the memory wasted.
Roxie uses a separate instance of IRowManager for each query. This provides the mechanism for limiting how much memory a query uses. Thor uses a single instance of an IRowManager for each slave/master.
Memory is allocated from a set of "heaps - where each heap allocates blocks of memory of a single size. The heap exclusively owns a set of heaplet (each 1 page in size), which are held in a doubly linked list, and sub allocates memory from those heaplets.
Information about each heaplet is stored in the base of the page (using a class with virtual functions) and the address of an allocated row is masked to determine which heap object it belongs to, and how it should be linked/released etc. Any pointer not in the allocated virtual address (e.g., constant data) can be linked/released with no effect.
Each heaplet contains a high water mark of the address within the page that has already been allocated (freeBase), and a lockless singly-linked list of rows which have been released (r_block). Releasing a row is non-blocking and does not involve any spin locks or critical sections. However, this means that empty pages need to be returned to the global memory pool at another time. (This is done in releaseEmptyPages()).
When the last row in a page is released a flag (possibleEmptyPages) is set in its associated heap. * This is checked before trying to free pages from a particular heap, avoiding waiting on a lock and traversing a candidate list.
Any page which might contain some spare memory is added to a lockless spare memory linked list. * Items are popped from this list when a heap fails to allocate memory from the current heaplet. Each item is checked in turn if it has space before allocating a new heaplet. * The list is also walked when checking to see which pages can be returned to the global memory. The doubly linked heaplet list allows efficient freeing.
Each allocation has a link count and an allocator id associated with it. The allocator id represents the type of the row, and is used to determine what destructor needs to be called when the row is destroyed. (The count for a row also contains a flag in the top bit to indicate if it is fully constructed, and therefore valid for the destructor to be called.)
A specialized heap is used to manage all allocations that require more than one page of memory. These allocations are not held on a free list when they are released, but each is returned directly to the global memory pool. Allocations in the huge heap can be expanded and shrunk using the resizeRow() functions - see below.
By default a fixed size heaps rounds the requested allocation size up to the next bucket size. A packed heap changes this behaviour and it is rounded up to the next 4 byte boundary instead. This reduces the amount of memory wasted for each row, but potentially increases the number of distinct heaps.
By default all fixed size heaps of the same size are shared. This reduces the memory consumption, but if the heap is used by multiple threads it can cause significant contention. If a unique heap is specified then it will not be shared with any other requests. Unique heaps store information about the type of each row in the heaplet header, rather than per row - which reduces the allocation overhead for each row. (Note to self: Is there ever any advantage having a heap that is unique but not packed??)
Blocked is an option on createFixedRowHeap() to allocate multiple rows from the heaplet, and then return the additional rows on subsequent calls. It is likely to reduce the average number of atomic operations required for each row being allocated, but the heap that is returned can only be used from a single thread because it is not thread safe.
By default the heaplets use a lock free singly linked list to keep track of rows that have been freed. This requires an atomic operation for each allocation and for each free. The scanning allocator uses an alternative approach. When a row is freed the row header is marked, and a row is allocated by scanning through the heaplet for rows that have been marked as free. Scanning uses atomic store and get, rather than more expensive synchronized atomic operations, so is generally faster than the linked list - provided a free row is found fairly quickly.
The scanning heaps have better best-case performance, but worse worse-case performance (if large numbers of rows need to be scanned before a free row is found). The best-case tends to be true if only one thread/activity is accessing a particular heap, and the worse-case if multiple activities are accessing a heap, particularly if the rows are being buffered. It is the default for thor which tends to have few active allocators, but not for roxie, which tends to have large numbers of allocators.
This is another varation on the scanning allocator, which further reduces the number of atomic operations. Usually when a row is allocated the link count on the heaplet is increased, and when it is freed the link count is decremented. This option delays decrementing the link count when the row is released, by marking the row with a different free flag. If it is subsequently reallocated there is no need to increment the link count. The downside is that it is more expensive to check whether a heaplet is completely empty (since you can no longer rely on the heaplet linkcount alone).
Thor has additional requirements to roxie. In roxie, if a query exceeds its memory requirements then it is terminated. Thor needs to be able to spill rows and other memory to disk and continue. This is achieved by allowing any process that stores buffered rows to register a callback with the memory manager. When more memory is required these callbacks are called to free up memory, and allow the job to continue.
Each callback can specify a priority - lower priority callbacks are called first since they are assumed to have a lower cost associated with spilling. When more memory is required the callbacks are called in priority order until one of them succeeds. The can also be passed a flag to indicate it is critical to force them to free up as much memory as possible.
A callback cannot allocate any memory from the memory manager. If it does it is likely to deadlock.
You cannot allocate memory while holding a lock if that lock is also required by a callback.
Again this will cause deadlock. If it proves impossible you can use a try-lock primitive in the callback, but it means you won't be able to spill those rows.
If the heaps are fragmented it may be more efficient to repack the heaps than spill to disk.
If you're resizing a potentially big block of memory use the resize function with the callback.
Some of the memory allocations cover more than one "page" - e.g., arrays used to store blocks of rows. (These are called huge pages internally, not to be confused with operating system support for huge pages...) When one of these memory blocks needs to be expanded you need to be careful:
Allocating a new page, copying, updating the pointer (within a cs) and then freeing is safe. Unfortunately it may involve copying a large chunk of memory. It may also fail if there isn't memory for the new and old block, even if the existing block could have been expanded into an adjacent block.
You can't lock, call a resize routine and update the pointer because the resize routine may need to allocate a new memory block- that may trigger a callback, which could in turn deadlock trying to gain the lock. (The callback may be from another thread...)
Therefore the memory manager contains a call which allows you to resize a block, but with a callback which is used to atomically update the pointer so it always remains thread safe.
Occasionally you have processes which read a large number of rows and then filter them so only a few are still held in memory. Rows tend to be allocated in sequence through the heap pages, which can mean those few remaining rows are scattered over many pages. If they could all be moved to a single page it would free up a significant amount of memory.
The memory manager contains a function to pack a set of rows into a smaller number of pages: IRowManager->compactRows().
This works by iterating through each of the rows in a list. If the row belongs to a heap that could be compacted, and isn't part of a full heaplet, then the row is moved. Since subsequent rows tend to be allocated from the same heaplet this has the effect of compacting the rows.
Much of the time Thor doesn't uses full memory available to it. If you are running multiple Thor processes on the same machine you may want to configure the system so that each Thor has a private block of memory, but there is also a shared block of memory which can be used by whichever process needs it.
The ILargeMemCallback provides a mechanism to dynamically allocate more memory to a process as it requires it. This could potentially be done in stages rather than all or nothing.
(Currently unused as far as I know... the main problem is that borrowing memory needs to be coordinated.)
When OS processes use a large amount of memory, mapping virtual addresses to physical addresses can begin to take a significant proportion of the execution time. This is especially true once the TLB is not large enough to store all the mappings. Huge pages can significantly help with this problem by reducing the number of TLB entries needed to cover the virtual address space. The memory manager supports huge pages in two different ways:
Huge pages can be preallocated (e.g., with hugeadm) for exclusive use as huge pages. If huge pages are enabled for a particular engine, and sufficient huge pages are available to supply the memory for the memory manager, then they will be used.
Linux kernels from 2.6.38 onward have support for transparent huge pages. These do not need to be preallocated, instead the operating system tries to use them behind the scenes. HPCC version 5.2 and following takes advantage of this feature to significantly speed memory access up when large amounts of memory are used by each process.
Preallocated huge pages tend to be more efficient, but they have the disadvantage that the operating system currently does not reuse unused huge pages for other purposes e.g., disk cache.
There is also a memory manager option to not return the memory to the operating system when it is no longer required. This has the advantage of not clearing the memory whenever it is required again, but the same disadvantage as preallocated huge pages that the unused memory cannot be used for disk cache. We recommend this option is selected when preallocated huge pages are in use - until the kernel allows them to be reused.
Changes in 6.x allow Thor to run multiple channels within the same process. This allows data that is constant for all channels to be shared between all slave channels - a prime example is the rhs of a lookup join. For the queries to run efficiently the memory manager needs to ensure that each slave channel has the same amount of memory - especially when memory is being used that is shared between them.
createGlobalRowManager() allows a single global row manager to be created which also provides slave row managers for the different channels via the querySlaveRowManager(unsigned slave) method.
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/Metrics.html b/devdoc/Metrics.html
new file mode 100644
index 00000000000..2b178d8b0b4
--- /dev/null
+++ b/devdoc/Metrics.html
@@ -0,0 +1,40 @@
+
+
+
+
+
+ Metrics Framework Design | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
This document describes the design of a metrics framework that allows HPCC Systems components to implement a metric collection strategy. Metrics provide the following functionality:
Alerts and monitoring
An important DevOps function is to monitor the cluster and providing alerts when problems are detected. Aggregated metric values from multiple sources provide the necessary data to build a complete picture of cluster health that drives monitoring and alerts.
Scaling
As described above, aggregated metric data is also used to dynamically respond to changing cluster demands and load. Metrics provide the monitoring capability to react and take action
Fault diagnosis and resource monitoring
Metrics provide historical data useful in diagnosing problems by profiling how demand and usage patterns may change prior to a fault. Predictive analysis can also be applied.
Analysis of jobs/workunits and profiling
With proper instrumentation, a robust dynamic metric strategy can track workunit processing. Internal problems with queries should be diagnosed from deep drill down logging.
The document consists of several sections in order to provide requirements as well as the design of framework components.
: A measurement defined by a component that represents an internal state that is useful in a system reliability engineering function. In the context of the framework, a metric is an object representing the above.
Metric Value
: The current value of a metric.
Metric Updating
: The component task of updating metric state.
Collection
: A framework process of selecting relevant metrics based on configuration and then retrieving their values.
Reporting
: A framework process of converting values obtained during a collection into a format suitable for ingestion by a collection system.
Trigger
: What causes the collection of metric values.
Collection System
: The store for metric values generated during the reporting framework process.
Roxie desires to keep a count of many different internal values. Some examples are
Disk type operations such as seeks and reads
Execution totals
Need to track items such as total numbers of items such as success and failures as well as breaking some counts into individual reasons. For example, failures may need be categorized such as as
Busy
Timeout
Bad input
Or even by priority (high, low, sla, etc.)
Current operational levels such as the length of internal queues
The latency of operations such as queue results, agent responses, and gateway responses
Roxie also has the need to track internal memory usage beyond the pod/system level capabilities. Tracking the state of its large fixed memory pool is necessary.
The Roxie buddy system also must track how often and who is completing requests. The "I Beat You To It" set of metrics must be collected and exposed in order to detect pending node failure. While specific action on these counts is not known up front, it appears clear that these values are useful and should be collected.
There does not appear to be a need for creating and destroying metrics dynamically. The set of metrics is most likely to be created at startup and remain active through the life of the Roxie. If, however, stats collection seeps into the metrics framework, dynamic creation and destruction of stats metrics is a likely requirement.
There are some interesting decisions with respect to ESP and collection of metrics. Different applications within ESP present different use cases for collection. Ownership of a given task drives some of these use cases. Take workunit queues. If ownership of the task, with respect to metrics, is WsWorkunits, then use cases are centric to that component. However, if agents listening on the queue are to report metrics, then a different set of use cases emerge. It is clear that additional work is needed to generate clear ownership of metrics gathered by ESP and/or the tasks it performs.
ESP needs to report the activeTransactions value from the TxSummary class(es). This gives an indication of how busy the ESP is in terms of client requests.
Direct measurement of response time in requests may not be useful since the type of request causes different execution paths within ESP that are expected to take widely varying amounts of time. Creation of metrics for each method is not recommended. However, two possible solutions are to a) create a metric for request types, or b) use a histogram to measure response time ranges. Another option mentioned redefines the meaning of a bucket in a histogram. Instead of a numeric distribution, each bucket represents a unique subtask within an overall "metric" representing a measured operation. This should be explored whether for operational or developmental purposes.
For tracking specific queries and their health, the feeling is that logging can accomplish this better than metrics since the list of queries to monitor will vary between clusters. Additionally, operational metrics solving the cases mentioned above will give a view into the overall health of ESP which will affect the execution of queries. Depending on actions taken by these metrics, scaling may solve overload conditions to keep cluster responsiveness acceptable.
For Roxie a workunit operates as a service. Measuring service performance using a histogram to capture response times as a distribution may be appropriate. Extracting the 95th percentile of response time may be useful as well.
There are currently no use cases requiring consistency between values of different metrics.
At this time the only concrete metric identified is the number of requests received. As the framework design progresses and ESP is instrumented, the list will grow.
This section covers the design and architecture of the framework. It discusses the main areas of the design, the interactions between each area, and an overall process model of how the framework operates.
The framework consists of three major areas: metrics, sinks, and the glue logic. These areas work together with the platform and the component to provide a reusable metrics collection function.
Metrics represent the quantifiable component state measurements used to track and assess the status of the component. Metrics are typically scalar values that are easily aggregated by a collection system. Aggregated values provide the necessary input to take component and cluster actions such as scaling up and down. The component is responsible for creating metrics and instrumenting the code. The framework provides the support for collecting and reporting the values. Metrics provide the following:
Simple methods for the component to update the metric
Simple methods for the framework to retrieve metric value(s)
Handling of all synchronization between updating and retrieving metric values
In addition, the framework provides the support for retrieving values so that the component does not participate in metric reporting. The component simply creates the metrics it needs, then instruments the component to update the metric whenever its state changes. For example, the component may create a metric that counts the total number of requests received. Then, wherever the component receives a request, a corresponding update to the count is added. Nowhere in the component is any code added to retrieve the count as that is handled by the framework.
Sinks provide a pluggable interface to hide the specifics of collection systems so that the metrics framework is independent of those dependencies. Sinks:
Operate independently of other sinks in the system
Convert metric native values into collection system specific measurements and reports
Drive the collection and reporting processes
The third area of the framework is the glue logic, referred to as the MetricsManager. It manages the metrics system for the component. It provides the following:
Handles framework initialization
Loads sinks as required
Manages the list of metrics for the component
Handles collection and reporting with a set of convenience methods used by sinks
The framework is designed to be instantiated into a component as part of its process and address space. All objects instantiated as part of the framework are owned by the component and are not shareable with any other component whether local or remote. Any coordination or consistency requirements that may arise in the implementation of a sink shall be the sole responsibility of the sink.
Components use metrics to measure their internal state. Metrics can represent everything from the number of requests received to the average length some value remains cached. Components are responsible for creating and updating metrics for each measured state. The framework shall provide a set of metrics designed to cover the majority of component measurement requirements. All metrics share a common interface to allow the framework to manage them in a common way.
To meet the requirement to manage metrics independent of the underlying metric state, all metrics implement a common interface. All metrics then add their specific methods to update and retrieve internal state. Generally the component uses the update method(s) to update state and the framework uses retrieval methods to get current state when reporting. The metric insures synchronized access.
For components that already have an implementation that tracks a metric, the framework provides a way to instantiate a custom metric. The custom metric allows the component to leverage the existing implementation and give the framework access to the metric value for collection and reporting. Note that custom metrics only support simple scalar metrics such as a counter or a gauge.
The framework defines a sink interface to support the different requirements of collection systems. Examples of collection systems are Prometheus, Datadog, and Elasticsearch. Each has different requirements for how and when measurements are ingested. The following are examples of different collection system requirements:
Polled vs Periodic
Single measurement vs multiple reports
Report format (JSON, text, etc.)
Push vs Pull
Sinks are responsible for two main functions: initiating a collection and reporting measurements to the collection system. The Metrics Reporter provides the support to complete these functions.
The sink encapsulates all of the collection system requirements providing a pluggable architecture that isolates components from these differences. The framework supports multiple sinks concurrently, each operating independently.
Instrumented components are not aware of the sink or sinks in use. Sinks can be changed without requiring changes to a component. Therefore, components are independent of the collection system(s) in use.
The metrics reporter class provides all of the common functions to bind together the component, the metrics it creates, and the sinks to which measurements are reported. It is responsible for the following:
Initialization of the framework
Managing the metrics created by the component
Handling collection and reporting as directed by configured sinks
A counter metric is a monotonically increasing value that "counts" the total occurrences of some event. Examples include the number of requests received, or the number of cache misses. Once created, the component instruments the code with updates to the count whenever appropriate.
A gauge metric is a continuously updated value representing the current state of an interesting value in the component. For example, the amount of memory used in an internal buffer, or the number of requests waiting on a queue. A gauge metric may increase or decrease in value as needed. Reading the value of a gauge is a stateless operation in that there are no dependencies on the previous reading. The value returned shall always be the current state.
Once created, the component shall update the gauge anytime the state of what is measured is updated. The metric shall provide methods to increase and decrease the value. The sink reads the value during collection and reporting.
A custom metric is a class that allows a component to leverage existing metrics. The component creates an instance of a custom metric (a templated class) and passes a reference to the underlying metric value. When collection is performed, the custom metric simply reads the value of the metric using the reference provided during construction. The component maintains full responsibility for updating the metric value as the custom metric class provides no update methods. The component is also responsible for ensuring atomic access to the value if necessary.
Records counts of measurements according to defined bucket limits. When created, the caller defines as set of bucket limits. During event recording, the component records measurements. The metric separates each recorded measurement into its bucket by testing the measurement value against each bucket limit using a less than or equal test. Each bucket contains a count of measurements meeting that criteria. Additionally, the metric maintains a default bucket for measurements outside of the maximum bucket limit. This is sometimes known as the "inf" bucket.
Some storage systems, such as Prometheus, require each bucket to accumulate its measurements with the previous bucket(s). It is the responsibility of the sink to accumulate values as needed.
A histogram metric that allows setting the bucket limit units in one domain, but take measurements in another domain. For example, the bucket limits may represent millisecond durations, yet it is more effecient to use execution cycles to take the measurements. A scaled histogram converts from the the measurement domain (cycles) to the limit units domain using a scale factor provided at initialization. All conversions are encapsulated in the scaled histogram class such that no external scaling is required by any consumer such as a sink.
This section discusses configuration. Since Helm charts are capable of combining configuration data at a global level into a component's specific configuration, The combined configuration takes the form as shown below. Note that as the design progresses it is expected that there will be additions.
Where (based on being a child of the current component):
metrics
: Metrics configuration for the component
metrics.sinks
: List of sinks defined for the component (may have been combined with global config)
metrics.sinks[].type
: The type for the sink. The type is substituted into the following pattern to determine the lib to load: libhpccmetrics<type><shared_object_extension>
metrics.sinks[].name
: A name for the sink.
metrics.sinks[].settings
: A set of key/value pairs passed to the sink when initialized. It should contain information necessary for the operation of the sink. Nested YML is supported. Example settings are the prometheus server name, or the collection period for a periodic sink.
Metric names shall follow a convention as outlined in this section. Because different collection systems have different requirements for how metric value reports are generated, naming is split into two parts.
First, each metric is given a base name that describes what the underlying value is. Second, meta data is assigned to each metric to further qualify the value. For example, a set of metrics may count the number of requests a component has received. Each metric would have the same base name, but meta data would separate types of request (GET vs POST), or disposition such as pass or fail.
Meta data further qualifies a metric value. This allows metrics to have the same name, but different scopes or categories. Generally, meta data is only used to furher qualify metrics that would have the same base name, but need further distinction. An example best describes a use case for meta data. Consider a component that accepts HTTP requests, but needs to track GET and POST requests separately. Instead of defining metrics with names post_requests.received and get_requests.received, the component creates two metrics with the base name requests.received and attaches meta data describing the request type of POST to one and GET to the other.
Use of meta data allows aggregating both types of requests into a single combined count of received requests while allowing a breakdown by type.
Meta data is represented as a key/value pair and is attached to the metric by the component during metric creation. The sink is responsible for converting meta data into useful information for the collection system during reporting.
The Component Instrumentation section covers how meta data is added to a metric.
In order to instrument a component for metrics using the framework, a component must include the metrics header from jlib (jmetrics.hpp) and add jlib as a dependent lib (if not already doing so).
The general steps for instrumentation are
Create a metrics reporter object
Create metric objects for each internal state to measure and add each to the reporter
Add updates to each metric throughout the component wherever metric state changes
The metrics reporter is a singleton created using the platform defined singleton pattern template. The component must obtain a reference to the reporter. Use the following example:
cpp
using namespace hpccMetrics;
+MetricsManager &metricsManager = queryMetricsManager();
Metrics are wrapped by a standard C++ shared pointer. The component is responsible for maintaining a reference to each shared pointer during the lifetime of the metric. The framework keeps a weak pointer to each metric and thus does not maintain a reference. The following is an example of creating a counter metric and adding it to the reporter. The using namespace eliminates the need to prefix all metrics types with hpccMetrics. Its use is assumed for all code examples that follow.
Note the metric type for both the shared pointer variable and in the make_shared template that creates the metric and returns a shared pointer. Simply substitute other metric types and handle any differences in the constructor arguments as needed.
Once created, add updates to the metric state throughout the component code where required. Using the above example, the following line of code increments the counter metric by 1.
cpp
pCounter->inc(1);
Note that only a single line of code is required to update the metric.
That's it! There are no component requirements related to collection or reporting of metric values. That is handled by the framework and loaded sinks.
For convenience, there are function templates that handle creating the reporter, creating a metric, and adding the metric to the reporter. For example, the above three lines of code that created the reporter, a metric, and added it, can be replaced by the following:
auto pCount = createMetricAndAddToManager<CounterMetric>("metricName", "description");
+
For convenience a similar function template exists for creating custom metrics. For a custom metric the framework must know the metric type and have a reference to the underlying state variable. The following template function handles creating a custom metric and adding it to the reporter (which is created if needed as well):
auto pCustomMetric = createCustomMetricAndAddToManager("customName", "description", metricType, value);
+
Where:
metricType
A defined metric type as defined by the MetricType enum.
value
A reference to the underlying event state which must be a scalar value convertable to a 64bit unsigned integer (__uint64)
A component, depending on requirements, may attach meta data to further qualify created metrics. Meta data takes the form of key value pairs. The base metric class MetricBase constructor defines a parameter for a vector of meta data. Metric subclasses also define meta data as a constructor parameter, however an empty vector is the default. The IMetric interface defines a method for retrieving the meta data.
Meta data is order dependent.
Below are two examples of constructing a metric with meta data. One creates the vector and passes it as a parameter, the other constructs the vector in place.
Metric units are treated separately from the base name and meta data. The reason is to allow the sink to translate based on collection system requirements. The base framework provides a convenience method for converting units into a string. However, the sink is free to do any conversions, both actual units and the string representation, as needed.
Metric units are defined using a subset of the StaticsMeasure enumeration values defined in jstatscodes.h. The current values are used:
SMeasureTimeNs - A time measurement in nanoseconds
SMeasureCount - A count of events
SMeasureSize - Size in bytes
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/NewFileProcessing.html b/devdoc/NewFileProcessing.html
new file mode 100644
index 00000000000..565ffff3ad8
--- /dev/null
+++ b/devdoc/NewFileProcessing.html
@@ -0,0 +1,37 @@
+
+
+
+
+
+ Storage planes | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
YAML files. The following are the YAML definitions which are used to serialize file information from dali/external store to the engines and if necessary to the worker nodes.
This is already covered in the deployed helm charts. It has been extended and rationalized slightly.
storage:
: hostGroups: - name: <required> hosts: [ .... ] - name: <required> hostGroup: <name> count: <unsigned:#hosts> # how many hosts within the host group are used ?(default is number of hosts) offset: <unsigned:0> # index of first host included in the derived group delta: <unsigned:0> # first host within the range[offset..offset+count-1] in the derived group
planes:
+
+: name: \<required\> prefix: \<path\> \# Root directory for
+ accessing the plane (if pvc defined), or url to access plane.
+ numDevices: 1 \# number of devices that are part of the plane
+ hostGroup: \<name\> \# Name of the host group for bare metal
+ hosts: \[ host-names \] \# A list of host names for bare metal
+ secret: \<secret-id\> \# what secret is required to access the
+ files. options: \# not sure if it is needed
+
Changes: * The replication information has been removed from the storage plane. It will now be specified on the thor instance indicating where (if anywhere) files are replicated. * The hash character (#) in a prefix or a secret name will be substituted with the device number. This replaces the old includeDeviceInPath property. This allows more flexible device substition for both local mounts and storage accounts. The number of hashes provides the default padding for the device number. (Existing Helm charts will need to be updated to follow these new rules.) * Neither thor or roxie replication is special cased. They are represented as multiple locations that the file lives (see examples below). Existing baremetal environments would be mapped to this new representation with implicit replication planes. (It is worth checking the mapping to roxie is fine.)
file: - name: <logical-file-name> format: <type> # e.g. flat, csv, xml, key, parquet meta: <binary> # (opt) format of the file, (serialized hpcc field information). metaCrc: <unsigned> # hash of the meta numParts # How many file parts. singlePartNoSuffix: <boolean> # Does a single part file include .part_1_of_1? numRows: # total number of rows in the file (if known) rawSize: # total uncompressed size diskSize # is this useful? when binary copying? planes: [] # list of storage planes that the file is stored on. tlk: # ???Should the tlk be stored in the meta and returned? splitType: <split-format> # Are there associated split points, and if so what format? (And if so what variant?)
#options relating to the format of the input file:
: grouped: <boolean> # is the file grouped? compressed: <boolean> blockCompressed: <boolean> formatOptions: # Any options that relate to the file format e.g. csvTerminator. These are nested because they can be completely free format recordSize: # if a fixed size record. Not really sure it is useful
part: \# optional information about each of the file parts (Cannot
+implement virtual file position without this) - numRows: \<count\>
+\# number of rows in the file part rawSize: \<size\> \# uncompressed
+size of the file part diskSize: \<size\> \# size of the part on disk
+
# extra fields that are used to return information from the file lookup service
missing: <boolean> # true if the file could not be found external: <boolean> # filename of the form external:: or plane:
If the information needs to be signed to be passed to dafilesrv for example, the entire structure of (storage, files) is serialized, and compressed, and that then signed.
Logically executed on the engine, and retrived from dali or in future versions from an esp service (even if for remote reads).
GetFileInfomation(<logical-filename>, <options>)
The logical-filename can be any logical name - including a super file, or an implicit superfile.
options include: * Are compressed sizes needed? * Are signatures required? * Is virtual fileposition (non-local) required? * name of the user
This returns a structure that provides information about a list of files
meta:
: hostGroups: storage: files: secrets: #The secret names are known, how do we know which keys are required for those secrets?
Some key questions: * Should the TLK be in the dali meta information? [Possibly, but not in first phase. ] * Should the split points be in the dali meta information? [Probably not, but the meta should indicate whether they exist, and if so what format they are. ] * Super files (implicit or explicit) can contain the same file information more than once. Should it be duplicated, or have a flag to indicate a repeat. [I suspect this is fairly uncommon, so duplication would be fine for the first version.] * What storage plane information is serialized back? [ all is simplest. Can optimize later. ]
NOTE: This doesn't address the question of writing to a disk file...
Local class for interpreting the results. Logically executed on the manager, and may gather extra information that will be serialized to all workers. The aim is that the same class implementations are used by all the engines (and fileview in esp).
MasterFileCollection : RemoteFileCollection : FileCollection(eclReadOptions, eclFormatOptions, wuid, user, expectedMeta, projectedMeta); MasterFileCollection //Master has access to dali RemoteFileCollection : has access to remote esp // think some more
FileCollection::GatherFileInformation(<logical-filename>, gatherOptions); - potentially called once per query. - class is responsible for optimizing case where it matches the previous call (e.g. in a child query). - possibly responsible for retrieving the split points ()
Following options are used to control whether split points are retrieved when file information is gathered * number of channels reading the data? * number of strands reading each channel? * preserve order?
gatherOptions: * is it a temporary file?
This class serializes all information to every worker, where it is used to recereate a copy of the master filecollection. This will contain information derived from dali, and locally e.g. options specified in the activity helper. Each worker has a complete copy of the file information. (This is similar to dafilesrv with security tokens.)
The files that are actually required by a worker are calculated by calling the following function. (Note the derived information is not serialized.)
Things to bear in mind: - Optimize same file reused in a child query (filter likely to change) - Optimize same format reused in a child query (filename may be dynamic) - Intergrating third party file formats and distributed file systems may require extra information. - optimize reusing the format options. - ideally fail over to a backup copy midstream.. and retry in failed read e.g. if network fault
: planes: #Simple 400 way thor - name: thor400 prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group #The storage plane used for replicating files on thor. - name: thor400_R1 prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group offset: 1 # A 200 way thor using the first 200 nodes as the thor 400 - name: thor200A prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group size: 200 # A 200 way thor using the second 200 nodes as the thor 400 - name: thor200B prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group size: 200 start: 200 # The replication plane for a 200 way thor using the second 200 nodes as the thor 400 - name: thor200B_R1 prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group size: 200 start: 200 offset: 1 # A roxie storage where 50way files are stored on a 100 way roxie - name: roxie100 prefix: /var/lib/HPCCSystems/roxie100 hosts: thor400Group size: 50 # The replica of the roxie storage where 50way files are stored on a 100 way roxie - name: roxie100_R1 prefix: /var/lib/HPCCSystems/thor400 hosts: thor400Group start: 50 size: 50
There is no special casing of roxie replication, and each file exists on multiple storage planes. All of these should be considered when determining which is the best copy to read from a particular engine node.
Creating storage planes from an existing systems [implemented]
b) [a] Start simplifying information in dali meta (e.g. partmask, remove full path name) c) [a] Switch reading code to use storageplane away from using dali path and environment paths - in ALL disk reading and writing code - change numDevices so it matches the container d) [c] Convert dali information from using copies to multiple groups/planese) [a] Reimplement the current code to create an IPropertyTree from dali file information (in a form that can be reused in dali) *f) [e] Refactor existing PR to use data in an IPropertyTree and cleanly separate the interfaces. g) Switch hthor over to using the new classes by default and work through all issues h) Refactor stream reading code. Look at the spark interfaces for inspiration/compatibility i) Refactor disk writing code into common class? j) [e] create esp service for accessing meta information k) [h] Refactor and review azure blob code l) [k] Re-implement S3 reading and writing code.
m) Switch fileview over to using the new classes. (Great test they can be used in another context + fixes a longstanding bug.)
) Implications for index reading? Will they end up being treated as a normal file? Don't implement for 8.0, although interface may support it.
Buffer sizes: - storage plane specifies an optimal reading minimum - compression may have a requirement - the use for the data may impose a requirement e.g. a subset of the data, or only fetching a single record
parallel disk reading may want to read a big chunk, but then process in sections. groan.
Look at lambda functions to create split points for a file. Can we use the java classes to implement it on binary files (and csv/xml)?
****************** Reading classes and meta information****************** meta comes from a combination of the information in dfs and the helper
The main meta information uses the same structure that is return by the function that returns file infromation from dali. The format specific options are contained in a nested attribute so they can be completely arbitrary
The helper class also generates a meta structure. Some options fill in root elements - e.g. compressed. Some fill in a new section (hints: @x=y). The format options are generated from the paramaters to the dataset format.
note normally there is only a single (or very few) files, so merging isn't too painful. queryMeta() queryOptions() rename meta to format? ???
Where does DFUserver fit in in a container system?
DFU has the following main functionality in a bare metal system: a) Spray a file from a 1 way landing zone to an N-way thor b) Convert file format when spraying. I suspect utf-16->utf8 is the only option actually used. c) Spray multiple files from a landing zone to a single logical file on an N-way thor d) Copy a logical file from a remote environment e) Despray a logical file to an external landing zone. f) Replicate an existing logical file on a given group. g) Copy logical files between groups h) File monitoring i) logical file operations j) superfile operations
ECL has the ability to read a logical file directly from a landingzone using 'FILE::<ip>' file syntax, but I don't think it is used very frequently.
How does this map to a containerized system? I think the same basic operations are likely to be useful. a) In most scenarios Landing zones are likely to be replaced with (blob) storage accounts. But for security reasons these are likely to remain distinct from the main location used by HPCC to store datasets. (The customer will have only access keys to copy files to and from those storage accounts.) The containerized system has a way for ECL to directly read from a blob storage account ('PLANE::<plane'), but I imagine users will still want to copy the files in many situations to control the lifetime of the copies etc. b) We still need a way to convert from utf16 to utf8, or extend the platform to allow utf16 to be read directly. c) This is still equally useful, allowing a set of files to be stored as a single file in a form that is easy for ECL to process. d) Important for copying data from an existing bare metal system to the cloud, and from a cloud system back to a bare metal system. e) Useful for exporting results to customers f+g) Essentially the same thing in the cloud world. It might still be useful to have h) I suspect we will need to map this to cloud-specific apis. i+j) Just as applicable in the container world.
Broadly, landing zones in bare metal map to special storage planes in containerized, and groups also map to more general storage planes.
There are a couple of complications connected with the implementation:
Copying is currently done by starting an ftslave process on either the source or the target nodes. In the container world there is no local node, and I think we would prefer not to start a process in order to copy each file. 2) Copying between storage groups should be done using the cloud provider api, rather than transferring data via a k8s job.
Suggestions:
Have a load balanced dafilesrv which supports multiple replicas. It would have a secure external service, and an internal service for trusted components.
Move the ftslave logic into dafilesrv. Move the current code for ftslave actions into dafilesrv with new operations.
When copying from/to a bare metal system the requests are sent to the dafilesrv for the node that currently runs ftslave. For a container system the requests are sent to the loadbalanced service.
It might be possible to migrate to lamda style functions for some of the work...
A later optimization would use a cloud service where it was possible.
When local split points are supported it may be better to spray a file 1:1 along with partition information. Even without local split points it may still be better to spray a file 1:1 (cheaper).
What are the spray targets? It may need to be storage plane + number of parts, rather than a target cluster. The default number of parts is the #devices on the storage plane.
=> Milestones a) Move ftslave code to dafilesrv (partition, pull, push) [Should be included in 7.12.x stream to allow remote read compatibility?] b) Create a dafilesrv component to the helm charts, with internal and external services. c) use storage planes to determine how files are sprayed etc. (bare-metal, #devices) Adapt dfu/fileservices calls to take (storageplane,number) instead of cluster. There should already be a 1:1 mapping from existing cluster to storage planes in a bare-metal system, so this may not involve much work. [May also need a flag to indicate if ._1_of_1 is appended?] d) Select correct dafilesrv for bare-metal storage planes, or load balanced service for other. (May need to think through how remote files are represented.)
=> Can import from a bare metal system or a containerized system using command line??
: NOTE: Bare-metal to containerized will likely need push operations on the bare-metal system. (And therefore serialized security information) This may still cause issues since it is unlikely containerized will be able to pull from bare-metal. Pushing, but not creating a logical file entry on the containerized system should be easier since it can use a local storage plane definition.
e) Switch over to using the esp based meta information, so that it can include details of storage planes and secrets.
: [Note this would also need to be in 7.12.x to allow remote export to containerized, that may well be a step too far]
f) Add option to configure the number of file parts for spray/copy/despray g) Ensure that eclwatch picks up the list of storage planes (and the default number of file parts), and has ability to specify #parts.
Later: h) plan how cloud-services can be used for some of the copies i) investigate using serverless functions to calculate split points. j) Use refactored disk read/write interfaces to clean up read and copy code. k) we may not want to expose access keys to allow remote reads/writes - in which they would need to be pushed from a bare-metal dafilesrv to a containerized dafilesrv.
Other dependencies: * Refactored file meta information. If this is switching to being plane based, then the meta information should also be plane based. Main difference is not including the path in the meta information (can just be ignored) * esp service for getting file information. When reading remotely it needs to go via this now...
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/README.html b/devdoc/README.html
new file mode 100644
index 00000000000..7efeee778f6
--- /dev/null
+++ b/devdoc/README.html
@@ -0,0 +1,24 @@
+
+
+
+
+
+ Developer Documentation | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
This document covers security configuration values and meanings. It does not serve as the source for how to configure security, but rather what the different values mean. These are not covered in the docs nor does any reasonable help information exist in the config manager or yaml files.
Security is configured either through an LDAP server or a plugin. Additionally, these are supported in both legacy deployments that use environment.xml and containerized deployments using Kubernetes and Helm charts. While these methods differ, the configuration values remain the same. Focus is placed on the different values and not the deployment target. Differences based on deployment can be found in the relevant platform documents.
Security is implemented via a security manager interface. Managers are loaded and used by components within the system to check authorization and authentication. LDAP is an exception to the loadable manager model. It is not a compliant loadable module like other security plugins. For that reason, the configuration for each is separated into two sections below: LDAP and Plugin Security Managers.
LDAP is a protocol that connects to an Active Directory server (AD). The term LDAP is used interchangeably with AD. Below are the configuration values for an LDAP connection. These are valid for both legacy (environment.xml) and containerized deployments. For legacy deployments the configuration manager is the primary vehicle for setting these values. However, some values are not available through the tool and must be set manually in the environment.xml if needed for a legacy deployment.
In containerized environments, a LDAP configuration block is required for each component. Currently, this results in a verbose configuration where much of the information is repeated.
LDAP is capable if handling user authentication and feature access authorization (such as filescopes).
Value
Example
Meaning
adminGroupName
HPCCAdmins
Group name containing admin users for the AD
cacheTimeout
60
Timeout in minutes to keep cached security data
ldapCipherSuite
N/A
Used when AD is not up to date with latest SSL libs. AD admin must provide
ldapPort
389 (default)
Insecure port
ldapSecurePort
636 (default)
Secure port over TLS
ldapProtocol
ldap
ldap for insecure (default), using ldapPort ldaps for secure using ldapSecurePort
ldapTimeoutSec
60 (default 5 for debug, 60 otherwise)
Connection timeout to an AD before rollint to next AD
Appears to only be used for IPlanet type ADs, but may still be required
systemCommonName
hpccAdmin
AD username of user to proxy all AD operations
systemPassword
System user password
AD user password
ldapAdminSecretKey
none
Key for Kubernetes secrets (4) (5)
ldapAdminVaultId
none
Vault ID used to load system username and password (5)
ldapDomain
none
Appears to be a comma separated version of the AD domain name components (5)
ldapAddress
192.168.10.42
IP address to the AD
commonBasedn
DC=z0lpf,DC=onmicrosoft,DC=com
Overrides the domain retrieved from the AD for the system user (5)
templateName
none
Template used when adding resources (5)
authMethod
none
Not sure yet
Notes:
modulesBaseDn is the same as resourcesBaseDn The code looks for first for modulesBaseDn and if not found will search for resourcesBaseDn
Allowed values for serverType are ActiveDirectory, AzureActiveDirectory, 389DirectoryServer, OpenLDAP, Fedora389
For AzureAD, users are managed from the AD dashboard, not via ECLWatch or through LDAP
If present, ldapAdminVaultId is read and systemCommonName and systemPassword are read from the Kubernetes secrets store and not from the LDAP config values
Must be configured manually in the environment.xml in legacy environments
Plugin security managers are separate shared objects loaded and initialized by the system. The manager interface is passed to components in order to provide necessary security functions. Each plugin has its own configuration. HPCC components can be configured to use a plugin as needed.
This document covers user authentication, the process of verifying the identity of the user. Authorization is a separate topic covering whether a user should be allowed to perform a specific operation of access a specific resource.
Each supported security manager is covered.
Generally, when authentication is needed, the security manager client should call the ISecManagerauthenticateUser method. The method also allows the caller to detect if the user being authenticated is a superuser. Use of that feature is beyond the scope of this document. In practice, this method is rarely if ever called. User authentication is generally performed as part of authorization. This is covered in more detail below.
This section covers how each supported security manager handles user authentication. As stated above, the method authenticateUser is defined for this purpose. However, other methods also perform user authentication. The sections that follow describe in general how each security manager performs user authentication, whether from directly calling the authenticateUser method, or as an ancillary action taken when another method is called.
The LDAP security manager uses the configured Active Directory to authenticate users. Once authenticated, the user is added to the permissions cache, if enabled, to prevent repeated trips to the AD whenever an authentication check is required.
If caching is enabled, a lookup is done to see if the user is already cached. If so, the cached user authentication status is returned. Note that the cached status remains until either the cache time to live expires or is cleared either manually or through some other programmatic action.
If caching is not enabled, a request is sent to the AD to validate the user credentials.
In either case, if digital signatures are configured, the user is also digitally signed using the username. Digitally signing the user allows for quick authentication by validating the signature against the username. During initial authentication, if the digital signature exists, it is verified to provide a fast way to authenticate the user. If the signature is not verified, the user is marked as not authenticated.
Authentication status is stored in the security user object so that further checks are not necessary when the same user object is used in multiple calls to the security manager.
Authentication in the htpasswd manager does not support singularly authenticating the user without also authorizing resource access. See the special case for authentication with authorization below.
Regardless, the htpasswd manager authenticates users using the .htpasswd file that is installed on the cluster. It does so by finding the user in the file and verifying that the input hashed password matches the stored hashed password in file.
The single user security manager allows the definition of a single username with a password. The values are set in the environment configuration and are read during the initialization of the manager. All authentication requests validate against the configured username and password. The process is a simple comparison. Note that the password stored in the environment is hashed.
Since resource access authorization requires an authenticated user, the authorization process also authenticates the user before checking authorization. There are a couple of advantages to this
Two separate calls are not required to check authorization (one to verify the user and one to check authorization)
The caller can perform third party authorization for specific user access
The authenticate method, or any of its overloads or derivatives, accepts a resource or resource list and a user. These methods authenticate the user first before checking access to the specified resource.
ECL Watch uses user authentication during authorization during its log in process. Instead of first authenticating the user, it calls an authenticate method passing both the user and the necessary resources for which the user must have access in order to log into ECL Watch.
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/StyleGuide.html b/devdoc/StyleGuide.html
new file mode 100644
index 00000000000..3dbf5807ec4
--- /dev/null
+++ b/devdoc/StyleGuide.html
@@ -0,0 +1,74 @@
+
+
+
+
+
+ Coding conventions | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Everyone has their own ideas of what the best code formatting style is, but most would agree that code in a mixture of styles is the worst of all worlds. A consistent coding style makes unfamiliar code easier to understand and navigate.
In an ideal world, the HPCC sources would adhere to the coding standards described perfectly. In reality, there are many places that do not. These are being cleaned up as and when we find time.
Unlike most software projects around, HPCC has some very specific constraints that makes most basic design decisions difficult, and often the results are odd to developers getting acquainted with its code base. For example, when HPCC was initially developed, most common-place libraries we have today (like STL and Boost) weren't available or stable enough at the time.
Also, at the beginning, both C++ and Java were being considered as the language of choice, but development started with C++. So a C++ library that copied most behaviour of the Java standard library (At the time, Java 1.4) was created (see jlib below) to make the transition, if ever taken, easier. The transition never happened, but the decisions were taken and the whole platform is designed on those terms.
Most importantly, the performance constraints in HPCC can make no-brainer decisions look impossible in HPCC. One example is the use of traditional smart pointers implementations (such as boost::shared_ptr or C++'s auto_ptr), that can lead to up to 20% performance hit if used instead of our internal shared pointer implementation.
The last important point to consider is that some libraries/systems were designed to replace older ones but haven't got replaced yet. There is a slow movement to deprecate old systems in favour of consolidating a few ones as the elected official ways to use HPCC (Thor, Roxie) but old systems still could be used for years in tests or legacy sub-systems.
In a nutshell, expect re-implementation of well-known containers and algorithms, expect duplicated functionality of sub-systems and expect to be required to use less-friendly libraries for the sake of performance, stability and longevity.
We use the extension .cpp for C++ source files, and .h or .hpp for header files. Header files with the .hpp extension should be used for headers that are internal to a single library, while header files with the .h extension should be used for the interface that the library exposes. There will typically be one .h file per library, and one .hpp file per cpp file.
Source file names within a single shared library should share a common prefix to aid in identifying where they belong.
Header files with extension .ipp (i for internal) and .tpp (t for template) will be phased out in favour of the scheme described above.
We adopted a Java-like inheritance model, with macro substitution for the basic Java keywords. This changes nothing on the code, but make it clearer for the reader on what's the recipient of the inheritance doing with it's base.
interface (struct): declares an interface (pure virtual class)
extends (public): One interface extending another, both are pure virtual
implements (public): Concrete class implementing an interface
There is no semantic check, which makes it difficult to enforce such scheme, which has led to code not using it intermixed with code using it. You should use it when possible, most importantly on code that already uses it.
We also tend to write methods inline, which matches well with C++ Templates requirements. We, however, do not enforce the one-class-per-file rule.
See the Interfaces section for more information on our implementation of interfaces.
Class and interface names are in CamelCase with a leading capital letter. Interface names should be prefixed capital I followed by another capital. Class names may be prefixed with a C if there is a corresponding I-prefixed interface name, e.g. when the interface is primarily used to create an opaque type, but need not be otherwise.
Variables, function and method names, and parameters use camelCase starting with a lower case letter. Parameters may be prefixed with underscore when the parameter is used to initialize a member variable of the same name. Common cases are constructors and setter methods.
Use real pointers when you can, and smart pointers when you have to. Take extra care on understanding the needs of your pointers and their scope. Most programs can afford a few dangling pointers, but a high-performance clustering platform cannot.
Most importantly, use common sense and a lot of thought. Here are a few guidelines:
Use real pointers for return values, parameter passing.
For .md variables use real pointers if their lifetime is guaranteed to be longer than the function (and no exception is thrown from functions you call), shared pointers otherwise.
Use Shared pointers for member variables - unless there is a strong guarantee the object has a longer lifetime.
Create Shared<X> with either:
Owned<X>: if your new pointer will take ownership of the pointer
Linked<X>: if you are sharing the ownership (shared)
Warning: Direct manipulation of the ownership might cause Shared<> pointers to lose the pointers, so subsequent calls to it (like o2->doIt() after o3 gets ownership) will cause segmentation faults.
Refer to [Reference counted objects]{.title-ref} for more information on our smart pointer implementation, Shared<>.
Methods that return pointers to link counted objects, or that use them, should use a common naming standard:
Foo * queryFoo() Does not return a linked pointer since lifetime is guaranteed for a set period. Caller should link if it needs to retain it for longer.
Foo * getFoo() Returned value is linked and should be assigned to an owned, or returned directly.
void setFoo(Foo * x) Generally parameters to functions are assumed to be owned by the caller, the callee needs to link them if they are retained.
void setFoo(Foo * ownedX) Some calls do transfer ownership of parameters - the parameter should be named to indicate this. If the function only has a single signficant parameter then sometimes the name of the function indicates the ownership.
We use 4 spaces to indent each level. TAB characters should not be used.
The { that starts a new scope and the corresponding } to close it are placed on a new line by themselves, and are not indented. This is sometimes known as the Allman or ANSI style.
We generally believe in the philosophy that well written code is self-documenting. Comments are also encouraged to describe why something is done, rather than how - which should be clear from the code.
javadoc-formatted comments for classes and interfaces are being added.
The virtual keyword should be included on the declaration of all virtual functions - including those in derived classes, and the override keyword should be used on all virtual functions in derived classes.
While C++ does not have explicit support for interfaces (in the java sense), an abstract class with no data members and all functions pure virtual can be used in the same way.
Interfaces are pure virtual classes. They are similar concepts to Java's interfaces and should be used on public APIs. If you need common code, use policies (see below).
An interface's name must start with an 'I' and the base class for its concrete implementations should start with a 'C' and have the same name, ex:
cpp
CFoo : implements IFoo { };
When an interface has multiple implementations, try to stay as close as possible to this rule. Ex:
Or, for partial implementation, use something like this:
cpp
CFoo : implements IFoo { };
+CFooCool : public CFoo { };
+CFooWarm : public CFoo { };
Extend current interfaces only on a 'is-a' approach, not to aggregate functionality. Avoid pollution of public interfaces by having only the public methods on the most-base interface in the header, and internal implementation in the source file. Prefer pImpl idiom (pointer-to-implementation) for functionality-only requirements and policy based design for interface requirements.
Example 1: You want to decouple part of the implementation from your class, and this part does not implements the interface your contract requires.:
cpp
interface IFoo
+{
+ virtual void foo()=0;
+};
+
+// Following is implemented in a separate private file...
+class CFoo : implements IFoo
+{
+ MyImpl *pImpl;
+public:
+ virtual void foo() override { pImpl->doSomething(); }
+};
Example2: You want to implement the common part of one (or more) interface(s) in a range of sub-classes.:
Shared<> is an in-house intrusive smart pointer implementation. It is close to boost's intrusive_ptr. It has two derived implementations: Linked and Owned, which are used to control whether the pointer is linked when a shared pointer is created from a real pointer or not, respectively. Ex:
cpp
Owned<Foo> myFoo = new Foo; // Take owenership of the pointers
+Linked<Foo> anotherFoo = = myFoo; // Shared ownership
Shared<> is thread-safe and uses atomic reference count handled by each object (rather than by the smart pointer itself, like boost's shared_ptr).
This means that, to use Shared<>, your class must implement the Link() and Release() methods - most commonly by extending the CInterfaceOf<> class, or the CInterface class (and using the IMPLEMENT_IINTERFACE macro in the public section of your class declaration).
This interface controls how you Link() and Release() the pointer. This is necessary because in some inner parts of HPCC, the use of a "really smart" smart pointer would add too many links and releases (on temporaries, local variables, members, etc) that could add to a significant performance hit.
The CInterface implementation also include a virtual function beforeDispose() which is called before the object is deleted. This allows resources to be cleanly freed up, with the full class hierarchy (including virtual functions) available even when freeing items in base classes. It is often used for caches that do not cause the objects to be retained.
The modern tool used for generating all our official assets is the Github Actions build-asset workflow on the hpcc-systems/HPCC-Platform repository, located here. Developers and contributors can utilize this same workflow on their own forked repository. This allows developers to quickly create assets for testing changes and test for errors before the peer review process.
Build assets will generate every available project under the HPCC-Platform namespace. There currently is not an option to control which packages in the build matrix get generated. But most packages get built in parallel, and released after the individual matrix job is completed, so there is no waiting on packages you don't need. Exceptions to this are for packages that require other builds to complete, such as the ECLIDE.
Upon completion of each step and matrix job in the workflow, the assets will be output to the repositories tags tab. An example for the hpcc-systems user repository is hpcc-systems/HPCC-Platform/tags.
The build assets workflow requires several repository secrets be available on a developers machine in order to run properly. You can access these secrets and variables by going to the settings tab in your forked repository, and then clicking on the Secrets and Variables - Actions drop down under Security on the lefthand side of the settings screen.
Create a secret by clicking the green New Repository Secret button. The following secrets are needed;
LNB_ACTOR - Your Github username
LNB_TOKEN - Classic Github token for your user with LN repo access
DOCKER_USERNAME - Your docker.io username
DOCKER_PASSWORD - Your docker.io password
SIGNING_CERTIFICATE - pks12 self signed cert encoded to base64 for windows signing
SIGNING_CERTIFICATE_PASSPHRASE - passphrase for pks12 cert
SIGNING_SECRET - ssh-keygen private key for signing linux builds
SIGN_MODULES_KEYID - email used to generate key
SIGN_MODULES_PASSPHRASE - passphrase for private key
The build-asset workflow is kicked off by a tag being pushed to the developers HPCC-Platform repository. Before we push the tag to our HPCC-Platform repository, we will want to have other tags in place if we want LN and ECLIDE builds to function correctly. Suggested tag patterns are community_HPCC-12345-rc1 or HPCC-12345-rc1.
If you choose not to tag the LN and ECLIDE builds, the community builds will generate but errors will be thrown for any build utilizing the LN repository. ECLIDE will not even attempt a build unless you are also successfully building LN due to the dependency scheme we use. The 'Baremetal' builds are designed to generate our clienttools targets for windows-2022 and macos-12 distributions. These jobs contain both the COMMUNITY and LN builds. If the LN build is not tagged, the COMMUNITY section of the job will run, and the assets will be uploaded, but the job will fail when it tries to build LN.
If you choose to precede your Jira number with community_ then you must tag LN with internal_ and ECLIDE with eclide_. Otherwise just use the Jira tag in all three repositories.
Once the LN and ECLIDE repository tags have been created and pushed with the same base branch that your work is based on for the HPCC-Platform, then you are free to push the HPCC-Platform tag which will initiate the build process.
The summary of the build-asset workflow can then be viewed for progress, and individual jobs can be selected to check build outputs.
Assets from the workflow will be released into the corresponding tag location, either in the HPCC-Platform repository for all community based builds, or the LN repository for any builds containing proprietary plugins. Simply browse to the releases or tag tab of your repository and select the tag name you just built. The assets will show up there as the build completes. An example of this on the hpcc-systems repository is hpcc-systems/HPCC-Platform/releases.
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/VersionSupport.html b/devdoc/VersionSupport.html
new file mode 100644
index 00000000000..b9029549af2
--- /dev/null
+++ b/devdoc/VersionSupport.html
@@ -0,0 +1,24 @@
+
+
+
+
+
+ Current versions | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
We release a new version of the platform every 3 months. If there are major changes in functionality, or significant backward compatibility issues then it will be tagged as a new major version, otherwise a new minor version. We normally maintain 4 versions of the system, which means that each new release will typically be supported for a year. Once a new major or minor version has been tagged gold it should not have any changes that change the behavior of queries.
Which versions should changes be applied to? The following gives some examples of the types of changes and which version they would be most relevant to target.
"master":
New features.
Bug fixes that will change the semantics of existing queries or processes.
Refactoring.
Performance improvements (unless simple and safe)
"<current>":
Bug fixes that only change behavior where it previously crashes or had undefined behavior (If well defined but wrong need to have very strong justification to change.)
Fixes for race conditions (the behavior was previously indeterminate so less of an argument against it changing)
Data corruption fixes - on a case by case basis if they change existing query results.
Missing functionality that prevents features from working.
Changes for tech-preview work that only effect those who are using it.
Regressions.
Improvements to logging and error messages (possibly in "previous" if simple and added to help diagnose problems).
Occasional simple refactoring that makes up-merging simpler..
Changes to improve backward compatibility of new features. (E.g. adding an ignored syntax to the compiler.)
Performance improvements - if simple and safe
"<previous>":
Simple bug fixes that do not change behavior
Simple changes for missing functionality
Regressions with simple fixes (but care is needed if it caused a change in behavior)
Serious regressions
Complex security fixes
"<critical>" fixes only:
Simple security fixes
Complex security fixes if sufficiently serious
"<security>" fixes only:
Serious security fixes
Occasionally earlier branches will be chosen, (e.g. security fixes to even older versions) but they should always be carefully discussed (and documented).
We aim to produce new point releases once a week. The point releases will contain
a) Any changes to the code base for that branch. b) Any security fixes for libraries that are project dependencies. We will upgrade to the latest point release for the library that fixes the security issue. c) For the cloud any security fixes in the base image or the packages installed in that image.
If there are no changes in any of those areas for a particular version then a new point release will not be created.
If you are deploying a system to the cloud you have one of two options
a) Use the images that are automatically built and published as part of the build pipeline. This image is currently based on ubuntu 22.04 and contains the majority of packages users will require.
b) Use your own hardened base image, and install the containerized package that we publish into that image.
We currently generate the following versions of the package and images:
debug
release with symbols
release without symbols.
It is recommended that you deploy the "release with symbols" version to all bare-metal and non-production cloud deployments. The extra symbols allow the system to generate stack backtraces which make it much easier to diagnose problems if they occur. The "release without symbols" version is recommended for Kubernetes production deployments. Deploying a system without symbols reduces the size of the images. This reduces the time it takes Kubernetes to copy the image before provisioning a new node.
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/Workunits.html b/devdoc/Workunits.html
new file mode 100644
index 00000000000..dce738a29b3
--- /dev/null
+++ b/devdoc/Workunits.html
@@ -0,0 +1,426 @@
+
+
+
+
+
+ Understanding workunits | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
A workunit contains all the information that the system requires about a query - including the parameters it takes, how to execute it, and how to format the results. Understanding the contents of a workunit is a key step to understanding how the HPCC system fits together. This document begins with an overview of the different elements in a workunit. That is then followed by a walk-through executing a simple query, with a more detailed description of some of the workunit components to show how they all tie together.
Before looking at the contents of a workunit it is important to understand one of the design goals behind the HPCC system. The HPCC system logically splits the code that is needed to execute a query in two. On the one hand there are the algorithms that are used to perform different dataset operations (e.g. sorting, deduping). The same algorithms are used by all the queries that execute on the system. On the other hand there is the meta-data that describes the columns present in the datasets, which columns you need to sort by, and the order of operations required by the query. These are typically different for each query. This "meta-data" includes generated compare functions, classes that describe the record formats, serialized data and graphs.
A workunit only contains data and code relating to the meta data for the query, i.e. "what to do", while the different engines (hthor, roxie, and thor) implement the algorithms - "how to do it". If you look at a workunit for an ECL query that sorts a dataset you will not find code to perform the sort itself in the workunit - you will not even find a call to a sort library function - that logic is contained in the engine that executes the query.
One consequence of this split, which can be initially confusing, is that execution continually passes back and forth between the generated code and the engines. By the end of this document you should have a better understanding of how the generated code is structured and the part it plays in executing a query.
Note the term "Query" is used as a generic term to cover read-only queries (typically run in roxie) and ETL (Extract, Transform, and Load) style queries that create lots of persistent datafiles (typically run in Thor). Also, the term "workunit" is used ambiguously. The dll created from a query is called a workunit (which is static), but "workunit" is also used to describe a single execution of a query (which includes the parameters and results). It should be clear from the context which of these is meant.
Throughout this document "dll" is a generic term used to refer to a dynamically loaded library. These correspond to shared objects in Linux (extension '.so'), dynamic libraries in Max OS X ('.dylib'), and dynamic link libraries in windows ('.dll').
A workunit is generated by the ecl compiler, and consists of a single dll. That dll contains several different elements:
Various C++ helper classes, and exported factory methods are used to create instances of those classes.
An XML resource containing different pieces of information about a workunit, including workflow and graphs.
Other user-defined resources included in the manifest.
A workunit dll contains everything that the engines need to execute the query. When a workunit is executed, key elements of the xml information are cloned from the workunit dll and copied into a database. This is then augmented with other information as the query is executed - e.g. input parameters, results, statistics, etc.. The contents of the workunit are accessed through an "IWorkUnit" interface (defined in common/workunit/workunit.hpp) that hides the implementation details.
(Workunit information is currently stored in the Dali database - one of the components within the HPCC platform. Work is in-progress to allow the bulk of this workunit data to be stored in Cassandra or another third-party database instead.)
The workunit information is used by most of the components in the system. The following is a quick outline:
eclcc
Creates a workunit dll from an ecl query.
eclccserver
Executes eclcc to create a workunit dll, and then clones some of the information into dali to create an active instance, ready to execute.
esp
Uses information in the workunit dll to publish workunits. This includes details of the parameters that the query takes, how they should be formatted, and the results it returns.
eclscheduler
Monitors workunits that are waiting for events, and updates them when those events occur.
eclagent/Roxie
Process the different workflow actions, and workflow code.
hThor/Roxie/Thor
Execute graphs within the workflow items.
Dali
This database is used to store the state of the workunit state.
The following ECL will be used as an example for the rest of the discussion. It is a very simple search that takes a string parameter 'searchName', which is the name of the person to search for, and returns the matching records from an index. It also outputs the word 'Done!' as a separate result.
This section outlines the different sections in a workunit. This is followed by a walk-through of the stages that take place when a workunit is executed, together with a more detailed explanation of the workunit contents.
The workflow is the highest level of control within a workunit. It is used for two related purposes:
Scheduling The HPCC system allows ECL code to be executed when certain events occur - e.g. every hour or when files are uploaded to a landing zone. Each piece of ECL code that is triggered by an external event creates a separate workflow action. This allows each of those events to be processed independently.
Splitting up queries. There are situations where it is useful to break parts of an ECL query into independent sections. The simplest example is the PERSIST workflow operation, which allows results to be shared between different work units. Each workflow operation creates one (or sometimes more) independent workflow items, which are then connected together.
Each piece of independent ECL is given a unique workflow id (wfid). Often workflow items need to be executed in a particular order, e.g. ensuring a persist exists before using it, which is managed with dependencies between different workflow items.
Our example above generates the following XML entry in the workunit:
This contains two workflow items. The first workflow item (wfid=1) ensures that the stored value has a default value if it has not been supplied. The second item (with wfid=2) is the main code for the query. This has a dependency on the first workflow item because the stored variable needs to be intialised before it is executed.
The generated code contains a class instance that is used for executing the code associated with the workflow items. It is generated at the end of the main C++ module. E.g.:
cpp
struct MyEclProcess : public EclProcess {
+ virtual int perform(IGlobalCodeContext * gctx, unsigned wfid) {
+ ....
+ switch (wfid) {
+ case 1U:
+ ... code for workflow item 1 ...
+ case 2U:
+ ... code for workflow item 2 ...
+ break;
+ }
+ return 2U;
+ }
+};
The main element is a switch statement inside the perform() function that allows the workflow engines to execute the code associated with a particular workflow item.
There is also an associated factory function that is exported from the dll, and is used by the engines to create instances of the class:
Most of the work executing a query involves processing dataset operations, which are implemented as a graph of activities. Each graph is represented in the workunit as an xml graph structure (currently it uses the xgmml format). The graph xml includes details of which types of activities are required to be executed, how they are linked together, and any other dependencies.
This graph contains a single subgraph (node id=2) that contains two activities - an index read activity and an output result activity. These activities are linked by a single edge (id "2_0"). The details of the contents are covered in the section on executing graphs below.
Each activity has a corresponding class instance in the generated code, and a factory function for creating instances of that class:
cpp
struct cAc2 : public CThorIndexReadArg {
+ ... Implementation of the helper for activity #2 ...
+};
+extern "C" ECL_API IHThorArg * fAc2() { return new cAc2; }
+
+struct cAc3 : public CThorWorkUnitWriteArg {
+ ... Implementation of the helper for activity #3 ...
+};
+extern "C" ECL_API IHThorArg * fAc3() { return new cAc3; }
The helper class for an activity implements the interface that is required for that particular kind. (The interfaces are defined in rtl/include/eclhelper.hpp - further details below.)
The are several other items, detailed below, that are logically associated with a workunit. The information may be stored in the workunit dll or in various other location e.g. Dali, Sasha or Cassandra. It is all accessed through the IWorkUnit interface in common/workunit/workunit.hpp that hides the implementation details. For instance information generated at runtime cannot by definition be included in the workunit dll.
Many queries contain input parameters that modify their behaviour. These correspond to STORED definitions in the ECL. Our example contains a single string "searchName", so the workunit contains a single input parameter:
in our example there are two - the dataset of results and the text string "Done!". The values of the results for a query are associated with the workunit. (They are currently saved in dali, but this may change in version 6.0.)
Other statistics and timings created when running the query are stored in the runtime copy of the workunit. (Statistics for graph elements are stored in a different format from global statistics, but the IWorkUnit interface ensures the implementation details are hidden.)
It is possible to include other user-defined resources in the workunit dll - e.g. web pages, or dashboard layouts. I have to confess I do not understand them... ??Tony please provide some more information....!
Once a workunit has been compiled to a dll it is ready to be executed. Execution can be triggered in different ways, E.g.:
The ECL for a query is submitted to esp
A workunit entry, containing the ECL, is created in dali and added to an eclccserver queue.
An eclccserver instance removes the workunit form the queue, and compiles the ECL to a workunit dll.
The dali workunit entry is updated with the information from the workunit dll.
The dali workunit is added to the agent execution queue associated with the target cluster.
The associated engine (actually agentexec for hThor and Thor) pulls a query form the queue and executes it.
A query is submitted and published with a name. Another request is then submitted to execute this previously compiled query.
A workunit entry, containing the ECL, is created in dali and added to an eclccserver queue.
An eclccserver instance removes the workunit form the queue, and compiles the ECL to a workunit dll.
There is a 'query set' for each combination of query name and the target cluster. The new workunit dll is added to the appropriate query set, and marked as the current active implementation.
Later, a query that references a named query is submitted to esp.
The name and target cluster are mapped via the query set to the active implementation, and a workunit instance is created from the active workunit dll.
The workunit is added to a roxie or eclagentexec queue ready to be executed.
The associated engine pulls a query form the queue and executes it.
A query is compiled as a stand alone executable. The executable is then run.
eclcc is executed on the command line without the -shared command line option.
The resulting executable is run. The engine used to execute the query depends on the -platform parameter supplied to eclcc.
Most queries create persistent workunits in dali and then update those workunits with results as they are calculated, however for some roxie queries (e.g. in a production system) the execution workunits are transient.
The following walk-through details the main stages executing a query, and the effect each of the query elements has.
The system uses several inter-process queues to communicate between the different components in the system. These queues are implemented by dali. Components can subscribe to one or more queues, and receive notifications when entries are avaialable.
Some example queues are:
<cluster>.eclserver - workunits to be compiled
<cluster>.roxie - workunits to execute on roxie
<cluster>.thor - graphs to execute on thor
<cluster>.eclscheduler - workunits that need to wait for events
<cluster>.agent - workunits to be executed on hthor or thor.
dfuserver_queue - dfu workunits for sprays/file copies etc.
When a workunit is ready to be run, the workflow controls the flow of execution. The workflow engine (defined in common/workunit/workflow.cpp) is responsible for determining which workflow item should be executed next.
The workflow for Thor and hThor jobs is coordinated by eclagent, while roxie includes the workflow engine in its process. The eclscheduler also uses the workflow engine to process events and mark workflow items ready for execution.
eclagent, or roxie calls the createProcess() function from the workunit dll to create an instance of the generated workflow helper class, and passes it to the workflow engine. The workflow engine walks the workflow items to find any items that are ready to be executed (have the state "reqd" - i.e. required). If a required workflow item has dependencies on other child workflow items then those children are executed first. Once all dependencies have executed successfully the parent workflow item is executed. The example has the following workflow entries:
Item 2 has a state of "reqd", so it should be evaluated now. Item 2 has a dependency on item 1, so that must be evaluated first. This is achieved by calling MyEclProcess::perform() on the object that was previously created from the workunit dll, passing in wfid = 1. That will execute the following code:
cpp
switch (wfid) {
+ case 1U:
+ if (!gctx->isResult("searchname",4294967295U)) {
+ ctx->setResultString("searchname",4294967295U,5U,"Smith");
+ }
+ break;
+ break;
+}
This checks if a value has been provided for the input parameter, and if not assigns a default value of "Smith". The function returns control to the workflow engine. With the dependencies for wfid 2 now satisfied, the generated code for that workflow id is now executed:
Most of the work for this workflow item involves executing graph1 (by calling back into eclagent/roxie). However, the code also directly sets another result. This is fairly typical - the code inside MyProcess::perform often combines evaluating scalar results, executing graphs, and calling functions that cannot (currently) be called inside a graph (e.g. those involving superfile transactions).
Once all of the required workflow items are executed, the workunit is marked as completed. Alternatively, if there are workflow items that are waiting to be triggered by an event, the workunit will be passed to the scheduler, which will keep monitoring for events.
Note that most items in the xml workflow begin in the state WFStateNull. This means that it is valid to execute them, but they haven't been executed yet. Typically, only a few items begin with the state WFStateReqd.
There are various specialised types of workflow items - e.g. sequential, wait, independent, but they all follow the same basic approach of executing dependencies and then executing that particular item.
Most of the interesting work in an ECL query is done within a graph. The call ctx->executeGraph will either execute the graph locally (in the case of hthor and roxie), or add the workunit onto a queue (for Thor). Whichever happens, control will pass to that engine.
Each item mode/type can affect the dependency structure of the workflow:
Sequential/Ordered
The workflow structure for sequential and ordered is the same. An item is made to contain all of the actions in the statement. This is achieved by making each action a dependency of this item. The dependencies, and consequently the actions, must be executed in order. An extra item is always inserted before each dependency. This means that if other statements reference the same dependency, it will only be performed once.
Persist
When the persist workflow service is used, two items are created. One item contains the graphs that perform the expression defined in ECL. It also stores the wfid for the second item. The second item is used to determine whether the persist is up to date.
Condition (IF)
The IF function has either 2 or 3 arguments: the expression, the trueresult, and sometimes the falseresult. For each argument, a workflow item is created. These items are stored as dependencies to the condition, in the order stated above.
Contingency (SUCCESS/FAILURE)
When a contingency clause is defined for an attribute, the attribute item stores the wfid of the contingency. If both success and failure are used, then the item will store the wfid of each contingency. The contingency is composed of items, just like the larger query.
Recovery
When a workflow item fails, if it has a recovery clause, the item will be re-executed. The clause contains actions defined by the programmer to remedy the problem. This clause is stored differently to SUCCESS/FAILURE, in that the recovery clause is a dependency of the failed item. In order to stop the recovery clause from being executed like the other dependencies, it is marked with WFStateSkip.
Independent
This specifier is used when a programmer wants to common up code for the query. It prevents the same piece of code from being executed twice in different contexts. To achieve this, an extra item is inserted between the expression and whichever items depend on it. This means that although the attribute can be referenced several times, it will only be executed once.
All the engines (roxie, hThor, Thor) execute graphs in a very similar way. The main differences are that hThor and Thor execute a sub graph at a time, while roxie executes a complete graph as one. Roxie is also optimized to minimize the overhead of executing a query - since the same query tends to be run multiple times. This means that roxie creates a graph of factory objects and those are then used to create the activities. The core details are the same for each of them though.
Each graph (e.g. graph1) consists of 1 or more subgraphs (in the example above, node id=1). Each of those subgraphs contains 1 or more activities (node id=2, node id=3).
The xml for each activity might contain the following information:
A unique id (e.g. id="2").
The "kind" of the activity, e.g. <att name="_kind" value="77"/>. The value is an item from the enum ThorActivityKind in eclhelper.hpp.
The ECL that created the activity. E.g. <att name="ecl" value="...">
The identifier of the ecl definition. E.g. <att name="name" value="results"/>
Location (e.g. file, line number, column) of the original ECL. E.g. <att name="definition" value="workuniteg1.ecl(3,1)"/>
Meta information the code generator has deduced about the activity output. Examples include the record size, expected number of rows, sort order etc. E.g. <att name="recordSize" value="120"/>
Hints, which are used for fine control of options for a particular activity (e.g,, the number of threads to use while sorting).
Record counts and stats once the job has executed. (These are logically associated with the activities in the graph, but stored separately.)
Graphs also contain edges that can take one of 3 forms:
Edges within graphs
: These are used to indicate how the activities are connected. The source activity is used as the input to the target activity. These edges have the following format:
<edge id="<source-activity-id>_<output-count>" source="<source-activity-id>" target="<target-activity-id">
+
+There is only one edge in our example workunit: \<edge id="2\_0"
+source="2" target="3"/\>.
+
Edges between graphs
: These are used to indicate direct dependencies between activities. For instance there will be an edge connecting the activity that writes a spill file to the activity that reads it. These edges have the following format:
<edge id="<source-activity-id>_<target-activity-id>" source="<source-subgraph-id>" target="<target-subgraph-id>"
+ <att name="_sourceActivity" value="<source-activity-id>"/>
+ <att name="_targetActivity" value="<target-activity-id>"/>
+ </edge>
+
+Roxie often optimizes spilled datasets and treats these edges as
+equivalent to the edges between activities.
+
Other dependencies.
: These are similar to the edges between graphs, but they are used for values that are used within an activity. For instance one part of the graph may calculate the maximum value of a column, and another activity may filter out all records that do not match that maximum value. The format is the same as the edges between graphs except that the edge contains the following attribute:
<att name="_dependsOn" value="1"/>
+
Each activity in a graph also has a corresponding helper class instance in the generated code. (The name of the class is "cAc" followed by the activity number, and the exported factory method is "fAc" followed by the activity number.) Each helper class implements a specialised interface (derived from IHThorArg) - the particular interface is determined by the value of the "_kind" attribute for the activity.
The contents of file rtl/include/eclhelper.hpp is key to understanding how the generated code relates to the activities. Each kind of activity requires a helper class that implements a specific interface. The helpers allow the engine to tailor the generalised activity implementation to the the particular instance - e.g. for a filter activity whether a row should be included or excluded. The appendix at the end of this document contains some further information about this file.
The classes in the generated workunits are normally derived from base implementations of those interfaces (which are implemented in rtl/include/eclhelper_base.hpp). This reduces the size of the generated code by providing default implementations for various functions.
For instance the helper for the index read (activity 2) is defined as:
a) onCreate() - common to all activities. It is called by the engine when the helper is first created, and allows the helper to cache information that does not change - in this case the name that is being searched for. b) getFileName() - determines the name of the index being read. c) createSegmentMonitors() - defines which columns to filter, and which values to match against. d) transform() - create the record to return from the activity. It controls which columns should be included from the index row in the output. (In this case all.)
To execute a graph, the engine walks the activities in the graph xml and creates, in memory, a graph of implementation activities.
For each activity, the name of the helper factory is calculated from the activity number (e.g. fAc2 for activity 2). That function is imported from the loaded dll, and then called to create an instance of the generated helper class - in this case cAc2.
The engine then creates an instance of the class for implementing the activity, and passes the previously created helper object to the constructor. The engine uses the _kind attribute in the graph to determine which activity class should be used. E.g. In the example above activity 2 has a _kind of 77, which corresponds to TAKindexread. For an index-read activity roxie will create an instance of CRoxieServerIndexReadActivity. (The generated helper that is passed to the activity instance will implement IHThorIndexReadArg). The activity implementations may also extract other information from the xml for the activity - e.g. hints. Once all the activities are created the edge information is used to link inputs activities to output activities and add other dependencies.
Note: Any subgraph that is marked with <att name="rootGraph" value="1"/> is a root subgraph. An activity within a subgraph that has no outputs is called a 'sink' (and an activity without any inputs is called a 'source').
Executing a graph involves executing all the root subgraphs that it contains. All dependencies of the activities within the subgraph must be executed before a subgraph is executed. To execute a subgraph, the engine executes each of the sink activities on separate threads, and then waits for each of those threads to complete. Each sink activity lazily pulls input rows on demand from activities further up the graph, processes them and returns when complete.
(If you examine the code you will find that this is a simplification. The implementation for processing dependencies is more fine grained to ensure IF datasets, OUPUT(,UPDATE) and other ECL constructs are executed correctly.)
In our example the execution flows as follows:
Only a single root subgraph, so need to execute that.
The engine will execute activity 3 - the workunit-write activity (TAKworkunitwrite).
The workunit-write activity will start its input.
The index-read activity will call the helper functions to obtain the filename, resolve the index, and create the filter.
The workunit-write activity requests a row from its input.
The index-read finds the first row, and calls the helper's transform() method to create an output row.
The workunit-write activity persists the row to a buffer (using the serializer provided by the IOutputMetaData interface implemented by the class mx1).
Back to step 5, workunit-write reading a row from its input, until end of file is returned (notified as two consecutive NULL rows.
Workunit-write commits the results and finishes.
The execution generally switches back and forth between the code in the engines, and the members of the generated helper classes.
There are some other details of query execution that are worth highlighting:
Child Queries
: Some activities perform complicated operations on child datasets of the input rows. E.g. remove all duplicate people who are marked as living at this address. This will create a "child query" in the graph - i.e. a nested graph within a subgraph, which may be executed each time a new input row is processed by the containing activity. (The graph of activities for each child query is created at the same time as the parent activity. The activity instances are reinitialised and re-executed for each input row processed by the parent activity to minimise the create-time overhead.)
Other helper functions
: The generated code contains other functions that are used to describe the meta information for the rows processed within the graph. E.g. the following class describes the output from the index read activity:
This represents a fixed size row that occupies 120 bytes. The object
+returned by the queryTypeInfo() function provides information about
+the types of the fields:
+
I.e. a string column, length 40 called "name", followed by a
+string column, length 80 called "address". The interface
+IOutputMetaData in eclhelper.hpp is key to understanding how the
+rows are processed.
+
Inline dataset operations
: The rule mentioned at the start - that the generated code does not contain any knowledge of how to perform a particular dataset operation - does have one notable exception. Some operations on child datasets are very simple to implement, and more efficient if they are implemented using inline C++ code. (The generated code is smaller, and they avoid the overhead of setting up a child graph.) Examples include filtering and aggregating column values from a child dataset.
The full code in the different engines is more complicated than the simplified process outlined above, especially when it comes to executing dependencies, but the broad outline is the same.
More information on the work done in the code generator to create the workunit can be found in ecl/eclcc/DOCUMENTATION.rst.
The C++ code can be generated as a single C++ file or multiple files. The system defaults to multiple C++ files, so that they can be compiled in parallel (and to avoid problems some compilers have with very large files). When multipe C++ files are generated the metadata classes and workflow classes are generated in the main module, and the activities are generated in the files suffixed with a number. It may be easier to understand the generated code if it is in one place. In which case, compile your query with the option -fspanMultipleCpp=0. Use -fsaveCppTempFiles to ensure the C++ files are not deleted (the C++ files will appear as helpers in the workunit details).
: The interface that is used by the workflow engine to execute the different workflow items in the generated code.
ThorActivityKind
: This enumeration contains one entry for each activity supported by the engines.
ICodeContext
: This interface is implemented by the engine, and provides a mechanism for the generated code to call back into the engine. For example resolveChildQuery() is used to obtain a reference to a child query that can then be executed later.
IOutputMetaData
: This interface is used to describe any meta data associated with the data being processed by the queries.
IHThorArg
: The base interface for defining information about an activity. Each activity has an associated interface that is derived from this interface. E.g. each instance of the sort activity will have a helper class implementing IHThorSortArg in the generated query. There is normally a corresponding base class for each interface in eclhelper_base.hpp that is used by the generated code e.g. CThorSortArg.
ARowBuilder
: This abstract base class is used by the transform functions to reserve memory for the rows that are created.
IEngineRowAllocator
: Used by the generated code to allocate rows and rowsets. Can also be used to release rows (or call the global function rtlReleaseRow()).
IGlobalCodeContext
: Provides access to functions that cannot be called inside a graph - i.e. can only be called from the global workflow code. Most functions are related to the internal implementation of particular workflow item types (e.g. persists).
: An activity is the basic unit of dataset processing implemented by the engines. Each activity corresponds to a node in the thor execution graph. Instances of the activities are connnected together to create the graph.
dll
: A dynamically loaded library. These correspond to shared objects in Linux (extension '.so'), dynamic libraries in Max OS X ('.dylib'), and dynamic link libraries in windows ('.dll').
superfile
: A composite file which allows a collection of files to be treated as a single compound file.
The XML for a workunit can be viewed on the XML tag in eclwatch, or generated by compiling the ECL using the -wu option with eclcc. Alternatively eclcc -b -S can be used to generate the XML and the C++ at the same time (the output filenames are derived from the input name).
Contributing Documentation to the HPCC Systems Platform Project
This document is intended for anyone that wants to contribute documentation to our project. The first audience is platform developers, so we can streamline the process of documenting new features. However, these guidelines apply to anyone who wants to contribute to any of our documentation (Language Reference, Programmer’s Guide, etc.).
This set of guidelines should help you understand the information needed to create sufficient documentation for any new feature you add. The good news is that we are all here to help you and support your writing efforts. We will help by advising you along the way, and by reviewing and editing your submissions.
Documenting a New Software Feature--Required and Optional Components
When you create a new feature or function, clear documentation is crucial for both internal teams and external users. You worked hard on the feature, so it deserves proper notice and usage.
Contributions to the platform are always welcome, and we strongly encourage developers and users to contribute documentation.
You can contribute on many levels:
Developer Notes
End user “Readmes” in the form of MD files in the GitHub repository
Blogs
Formal documentation
Regardless of the form you are planning to deliver, here are the required and optional components to include in a document.
Tip: VS Code is very good at editing MD files. There is a built-in Preview panel available to be able to see the rendered form.
In addition, GitHub Copilot is MD-aware and can help you write and format. For example, you can ask the Copilot, “How can I align the content within the cells of my Markdown table?” GitHub copilot will show you the alignment options.
What it is: Briefly describe the feature's purpose and the problem it solves.
Why it matters: Explain the value proposition for users and the overall impact on the software.
Target audience: Specify who this feature is designed for (for example, all users or specific user roles).
Use Cases: Provide concrete examples of how a user might leverage this feature in a real-world scenario.
Installation and Configuration: Details on how to install and basic setup for use, if needed. This must include any system requirements or dependencies.
User Guide / Functionality
How it works: Provide a task-oriented, step-by-step guide for using the feature. If possible, include screenshots for visual learners.
Tips, Tricks, and Techniques: Explain any shortcuts or clever uses of the feature that may be non-obvious.
Inputs and Outputs: Detail the information users need to provide to the feature and the format of the results.
Error Handling: Explain what happens if users encounter errors and how to troubleshoot common issues.
Limitations and Considerations:
Limitations: Acknowledge any restrictions or boundaries associated with the feature's functionality.
Detailed configuration options: If the feature offers advanced settings or customizations, provide in-depth instructions for experienced users. This is similar to the way some options command line program's usage are only displayed using the verbose option.
API Reference (for technical audiences)
Technical specifications: For features with an API component, include detailed API reference documentation for developers integrating it into their applications.
FAQs
Frequently Asked Questions: Address any commonly anticipated user questions about the feature to pre-empt confusion.
Additional Resources
Links to related documentation: Include links to relevant documentation for features that interact with this new addition.
Tutorials: Consider creating tutorials for a more interactive learning experience.
Videos: Consider requesting that a video be made to provide a more interactive visual learning experience. You should provide a simple script or outline of what should be shown in the video.
Target your audience: Tailor the level of detail and technical jargon you use based on whether the documentation is for developers or end-users.
Clarity and Conciseness: Use clear, concise language and maintain a consistent structure for easy navigation. Always use present tense, active voice. Remember, you’re writing for users and programmers, not academics, so keep it simple and straightforward. See the HPCC Style Guide for additional guidance.
Visual Aids: Screenshots, diagrams, and flowcharts can significantly enhance understanding. A picture can communicate instantly what a thousand words cannot.
Maintain and Update: Regularly review and update documentation as the feature evolves or based on user feedback.
By following these guidelines and including the required and optional components, you can create comprehensive documentation that empowers users and streamlines the adoption of your new software feature.
The boundary between a developer's responsibilities and the documentation team’s responsibility is not cast in stone. However, there are some guidelines that can help you decide what your responsibility is. Here are some examples:
Changing the default value of a configuration setting
This typically needs a simple one or two word change in the area of the documentation where that setting is documented. However, the change could impact existing deployments or existing code and therefore it might also require a short write-up for the Red Book and/or Release Announcement. If the setting is used by both bare-metal and containerized, you should provide information about how the new setting is used in each of those deployments.
Adding or modifying a Language keyword, Standard Library function, or command line tool action
This needs some changes to existing documentation so the best way to provide the information is in a documentation Jira issue. If it s a new keyword, function, or action, a brief overview should be included. For a Standard Library function, the developer should update the Javadoc comment in the appropriate ECL file. For a command line tool change, the developer should update the Usage section of the code.
This is a candidate for either an MD file, a blog, or both. Since there should have been some sort of design specification document, that could easily be repurposed as a good start for this.
A feature/function that is only used internally to the system
Since this is information that is probably only of interest to other developers, a write-up in the form of an MD file in the repo is the best approach. If it affects end-users or operations, then a more formal document or a blog might be a good idea. If it affects existing deployments or existing code, then a Red Book notice might also be needed.
New tests are frequently added and the regression suite readme files should be updated at the same time. If the tests are noteworthy, we could add a mention in the Platform Release Notes.
In general, it makes sense to keep simple documentation near the code. For example, a document about ECL Agent should go in the ECLAgent folder. However, there are times where that is either not possible or a document may cover more than one component. In those cases, there are a few options as shown below.
devdoc: This is a general folder for any developer document.
devdoc/docs: This is a folder for documents about documentation.
devdoc/userdoc: This is a collection of docs targeted toward the end-user rather than developers.
This is primarily for informal documents aimed at end-users. This info can and should be incorporated into the formal docs. devdoc/userdoc/troubleshoot: Information related to troubleshooting particular components
devdoc/userdoc/azure: Useful information about Azure Cloud portal and cli
devdoc/userdoc/roxie: Useful information for running roxie
devdoc/userdoc/thor: Useful information for running thor
devdoc/userdoc/blogs: COMING SOON: Location and storage of original text for blogs. It also has docs with guidelines and instructions on writing Blogs
You can include your documentation with your code in a Pull Request or create a separate Jira and Pull Request for the documentation. This depends on the size of the code and doc. For a large project or change, a separate Pull request for the documentation is better. This might allow the code change to be merged faster.
For minor code changes, for example the addition of a parameter to an existing ECL keyword, you can request a documentation change in a Jira issue. You should provide sufficient details in the Jira.
For example, If you add an optional parameter named Foo, you should provide details about what values can be passed in through the Foo parameter and what those values mean. You should also provide the default value used if the parameter is omitted.
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/docs/DocTemplate.html b/devdoc/docs/DocTemplate.html
new file mode 100644
index 00000000000..678d2b6fc1a
--- /dev/null
+++ b/devdoc/docs/DocTemplate.html
@@ -0,0 +1,24 @@
+
+
+
+
+
+ Feature Documentation Template | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Use this template to create new documentation for a feature, function, or process. Not every feature will require all six sections. Delete the ones that are not applicaple.
We strive to maintain a consistent voice. These guidelines can help your writing match that voice.
Use present tense, active voice. Documentation should be you speaking directly to the reader. Simply tell them what to do.
This example sentence: "The user selects the file menu." is passive voice. You wouldn’t say it that way in a conversation.
You should use active voice wording, such as: "Select the file menu".
Similarly, instructions like these are active voice: "Press the button" or "Submit the file".
Documentation is you instructing the user. Just tell them what to do.
Be Brief and keep it simple. Be efficient with your words. Keep sentences short and concise. Keep paragraphs short as well. Use just a few sentences per paragraph. Use simple words wherever possible and try to avoid lengthy explanations.
Consistency: Be consistent. Use the same voice across all documents. Use the same term, use the same spelling, punctuation, etc. Follow the conventions in this guide.
When writing formal documentation that is more than a new feature announcement, do not refer to a new feature, or coming features as such, this does not hold up well over time.
Do not refer to how things were done in the past. For example, "in the past we had to do X-Y-Z steps and now we no longer have to. This ‘new feature’ can do it in one step, Z". This only adds potential confusion. Just instruct on exactly what needs to be done now, using words efficiently as possible, so for this example just say “perform step Z”
There are many terms specific to HPCC Systems® and the HPCC Systems platform. Use the following style guide for word usage and capitalization guidelines to use when referring to system components or other HPCC-specific entities. Maintain consistent usage throughout all docs.
Officially and legally the organizaion's name is HPCC Systems® and it is a registered trademark.
You should always refer to the platform as the HPCC Systems® platform and the registration mark ® should appear in the first and most prominent mention of the name.
While it is acceptable to use the ® anywhere in a document, it is required to be used in the first and most prominent mention - so the average reader will be aware. Any usage after that first and most prominent is optional.
Components and Tools
HPCC Systems Platform
Dali
Sasha
Thor
hThor
ROXIE
DFU Server (Distributed File Utility Server)
ESP Server (Enterprise Server Platform)
ESP Services
WsECL
ECL Watch
ECL Server
ECLCC Server
ECL Agent
ECL IDE
ECL Plug-in for Eclipse
ECL Playground
LDAP
dafilesrv
VS Code (no hyphen)
ECL Language Extension for VS Code
Configuration Manager (not ConfigMgr or ConfigManager)
Note: when referring to the startup command, use configmgr (always lowercase)
To “assure” a person of something is to make him or her confident of it.
According to the Associated Press style, to “ensure” that something happens is to make certain that it does, and to “insure” is to issue an insurance policy.
Other authorities, however, consider “ensure” and “insure” interchangeable. We prefer ensure when it is not talking about insurance.
This series of blog posts started life as a series of walk-throughs and brainstorming sessions at a team offsite. This series will look at adding a new activity to the system. The idea is to give an walk through of the work involved, to highlight the different areas that need changing, and hopefully encourage others to add their own activities. In parallel with the description in this blog there is a series of commits to the github repository that correspond to the different stages in adding the activity. Once the blog is completed, the text will also be checked into the source control tree for future reference.
The new activity is going to be a QUANTILE activity, which can be used to find the records that split a dataset into equal sized blocks. Two common uses are to find the median of a set of data (split into 2) or percentiles (split into 100). It can also be used to split a dataset for distribution across the nodes in a system. One hope is that the classes used to implement quantile in Thor can also be used to improve the performance of the global sort operation.
It may seem fatuous, but the first task in adding any activity to the system is to work out what that activity is going to do! You can approach this in an iterative manner - starting with a minimal set of functionality and adding options as you think of them - or start with a more complete initial design. We have used both approaches in the past to add capabilities to the HPCC system, but on this occasion we will be starting from a more complete design - the conclusion of our initial design discussion:
"What are the inputs, options and capabilities that might be useful in a QUANTILE activity?"
The discussion produced the following items:
Which dataset is being processed?
This is always required and should be the first argument to the activity.
How many parts to split the dataset into?
This is always required, so it should be the next argument to the activity.
Which fields are being used to order (and split) the dataset?
Again this is always required, so the list of fields should follow the number of partitions.
Which fields are returned?
Normally the input row, but often it would be useful for the output to include details of which quantile a row corresponds to. To allow this an optional transform could be passed the input row as LEFT and the quantile number as COUNTER.
How about first and last rows in the dataset?
Sometimes it is also useful to know the first and last rows. Add flags to allow them to be optionally returned.
How do you cope with too few input rows (including an empty input)?
After some discussion we decided that QUANTILE should always return the number of parts requested. If there were fewer items in the input they would be duplicated as appropriate. We should provide a DEDUP flag for the situations when that is not desired. If there is an empty dataset as input then the default (blank) row will be created.
Should all rows have the same weighting?
Generally you want the same weighting for each row. However, if you are using QUANTILE to split your dataset, and the cost of the next operation depends on some feature of the row (e.g., the frequency of the firstname) then you may want to weight the rows differently.
What if we are only interested in the 5th and 95th centiles?
We could optionally allow a set of values to be selected from the results.
There were also some implementation details concluded from the discussions:
How accurate should the results be?
The simplest implementation of QUANTILE (sort and then select the correct rows) will always produce accurate results. However, there may be some implementations that can produce an approximate answer more quickly. Therefore we could add a SKEW attribute to allow early termination.
Does the implementation need to be stable?
In other words, if there are rows with identical values for the ordering fields, but other fields not part of the ordering with different values, does it matter which of those rows are returned? Does the relative order within those matching rows matter?
The general principle in the HPCC system is that sort operations should be stable, and that where possible activities return consistent, reproducible results. However, that often has a cost - either in performance or memory consumption. The design discussion highlighted the fact that if all the fields from the row are included in the sort order then the relative order does not matter because the duplicate rows will be indistinguishable. (This is also true for sorts, and following the discussion an optimization was added to 5.2 to take advantage of this.) For the QUANTILE activity we will add an ECL flag, but the code generator should also aim to spot this automatically.
Returning counts of the numbers in each quantile might be interesting.
This has little value when the results are exact, but may be more useful when a SKEW is specified to allow an approximate answer, or if a dataset might have a vast numbers of duplicates. It is possibly something to add to a future version of the activity. For an approximate answer, calculating the counts is likely to add an additional cost to the implementation, so the target engine should be informed if this is required.
Is the output always sorted by the partition fields?
If this naturally falls out of the implementations then it would be worth including it in the specification. Initially we will assume not, but will revisit after it has been implemented.
After all the discussions we arrived at the following syntax:
QUANTILE(<dataset>, <number-of-ranges>, { sort-order } [, <transform>(LEFT, COUNTER)]
+ [,FIRST][,LAST][,SKEW(<n>)][,UNSTABLE][,SCORE(<score>)][,RANGE(set)][,DEDUP][,LOCAL]
+
+FIRST - Match the first row in the input dataset (as quantile 0)
+LAST - Match the last row in the input dataset (as quantile <n>)
+SKEW - The maximum deviation from the correct results allowed. Defaults to 0.
+UNSTABLE - Is the order of the original input values unimportant?
+SCORE - What weighting should be applied for each row. Defaults to 1.
+RANGE - Which quantiles should actually be returned. (Defaults to ALL).
+DEDUP - Avoid returning a match for an input row more than once.
+
We also summarised a few implementation details:
The activity needs to be available in GLOBAL, LOCAL and GROUPED variants.
The code generator should derive UNSTABLE if no non-sort fields are returned.
Flags to indicate if a score/range is required.
Flag to indicate if a transform is required.
Finally, deciding on the name of the activity took almost as long as designing it!
The end result of this process was summarised in a JIRA issue: https://track.hpccsystems.com/browse/HPCC-12267, which contains details of the desired syntax and semantics. It also contains some details of the next blog topic - test cases.
Incidentally, a question that arose from of the design discussion was "What ECL can we use if we want to annotate a dataset with partition points?". Ideally the user needs a join activity which walks through a table of rows, and matches against the first row that contains key values less than or equal to the values in the search row. There are other situations where that operation would also be useful. Our conclusion was that the system does not have a simple way to achieve that, and that it was a deficiency in the current system, so another JIRA was created (see https://track.hpccsystems.com/browse/HPCC-13016). This is often how the design discussions proceed, with discussions in one area leading to new ideas in another. Similarly we concluded it would be useful to distribute rows in a dataset based on a partition (see https://track.hpccsystems.com/browse/HPCC-13260).
When adding new features to the system, or changing the code generator, the first step is often to write some ECL test cases. They have proved very useful for several reasons:
Developing the test cases can help clarify issues, and other details that the implementation needs to take into account. (E.g., what happens if the input dataset is empty?)
They provide something concrete to aim towards when implementing the feature.
They provide a set of milestones to show progress.
They can be used to check the implementation on the different engines.
As part of the design discussion we also started to create a list of useful test cases (they follow below in the order they were discussed). The tests perform varying functions. Some of the tests are checking that the core functionality works correctly, while others check unusual situations and that strange boundary cases are covered. The tests are not exhaustive, but they are a good starting point and new tests can be added as the implementation progresses.
The following is the list of tests that should be created as part of implementing this activity:
Compare with values extracted from a SORT. Useful to check the implementation, but also to ensure we clearly define which results we are expecting.
QUANTILE with a number-of-ranges = 1, 0, and a very large number. Should also test the number of ranges can be dynamic as well as a constant.
Empty dataset as input.
All input entries are duplicates.
Dataset smaller than number of ranges.
Input sorted and reverse sorted.
Normal data with small number of entries.
Duplicates in the input dataset that cause empty ranges.
Random distribution of numbers without duplicates.
Local and grouped cases.
SKEW that fails.
Test scoring functions.
Testing different skews that work on the same dataset.
An example that uses all the keywords.
Examples that do and do not have extra fields not included in the sort order. (Check that the unstable flag is correctly deduced.)
Globally partitioned already (e.g., globally sorted). All partition points on a single node.
Apply quantile to a dataset, and also to the same dataset that has been reordered/distributed. Check the resulting quantiles are the same.
Calculate just the 5 and 95 centiles from a dataset.
Check a non constant number of splits (and also in a child query where it depends on the parent row).
A transform that does something interesting to the sort order. (Check any order is tracked correctly.)
Check the counts are correct for grouped and local operations.
Call in a child query with options that depend on the parent row (e.g., num partitions).
Split points that fall in the middle of two items.
No input rows and DEDUP attribute specified.
Ideally any test cases for features should be included in the runtime regression suite, which is found in the testing/regress directory in the github repository. Tests that check invalid syntax should go in the compiler regression suite (ecl/regress). Commit https://github.com/ghalliday/HPCC-Platform/commit/d75e6b40e3503f851265670a27889d8adc73f645 contains the test cases so far. Note, the test examples in that commit do not yet cover all the cases above. Before the final pull request for the feature is merged the list above should be revisited and the test suite extended to include any missing tests.
In practice it may be easier to write the test cases in parallel with implementing the parser -since that allows you to check their syntax. Some of the examples in the commit were created before work was started on the parser, others during, and some while implementing the feature itself.
The first stage in implementing QUANTILE will be to add it to the parser. This can sometimes highlight issues with the syntax and cause revisions to the design. In this case there were two technical issues integrating the syntax into the grammar. (If you are not interested in shift/reduce conflicts you may want to skip a few paragraphs and jump to the walkthrough of the changes.)
Originally, the optional transform was specified inside an attribute, e.g., something like OUTPUT(transform). However, this was not very consistent with the way that other transforms were implemented, so the syntax was updated so it became an optional transform following the partition field list.
When the syntax was added to the grammar we hit another problem: Currently, a single production (sortList) in the grammar is used for matching sort orders. As well as accepting fields from a dataset the sort order production has been extended to accept any named attribute that can follow a sort order (e.g., LOCAL). This is because (with one token lookahead) it is ambiguous where the sort order finishes and the list of attributes begins. Trying to include transforms in those productions revealed other problems:
If a production has a sortList followed by a transform (or attribute) then it introduces a shift/reduce error on ','. To avoid the ambiguity all trailing attributes or values need to be included in the sortList.
Including a transform production in the sortList elements causes problems with other transform disambiguation (e.g., DATASET[x] and AGGREGATE).
We could require an attribute around the transform e.g., OUTPUT(transform), but that does not really fit in with other activities in the language.
We could change the parameter order, e.g., move the transform earlier, but that would make the syntax counter-intuitive.
We could require { } around the list - but this is inconsistent with some of the other sort orders.
In order to make some progress I elected to choose the last option and require the sort order to be included in curly braces. There are already a couple of activities - subsort and a form of atmost that similarly require them (and if redesigning ECL from scratch I would be tempted to require them everywhere). The final syntax is something that will need further discussion as part of the review of the pull request though, and may need to be revisited.
The ECL query is represented by a graph of "expression" nodes - each has a "kind" that comes from the enumeration _node_operator. The first requirement is to add a new enumeration to represent the new activity - in this case we elected to reuse an unused placeholder. (These placeholders correspond to some old operators that are no longer supported. They have not been removed because the other elements in the enumeration need to keep the same values since they are used for calculating derived persistent values e.g., the hashes for persists.)
New attribute names in hqlatoms.
The quantile activity introduces some new attribute names that have not been used before. All names are represented in an atom table, so the code in hqlatoms.hpp/cpp is updated to define the new atoms.
Properties of no_quantile
There are various places that need to be updated to allow the system to know about the properties of the new operator:
hqlattr
This contains code to calculate derived attributes. The first entry in the case statement is currently unused (the function should be removed). The second, inside calcRowInformation(), is used to predict how many rows are generated by this activity. This information is percolated through the graph and is used for optimizations, and input counts can be used to select the best implementation for a particular activity.
hqlexpr
Most changes are relatively simple including the text for the operator, whether it is constant, and the number of dataset arguments it has. One key function is getChildDatasetType() that indicates the kind of dataset arguments the operator has, which in turn controls how LEFT/RIGHT are interpreted. In this case some of the activity arguments (e.g., the number of quantiles) implicitly use fields within the parent dataset, and the transform uses LEFT, so the operator returns childdataset_datasetleft.
hqlir
This entry is used for generating an intermediate representation of the graph. This can be useful for debugging issues. (Running eclcc with the logging options "--logfile xxx" and "--logdetail 999" will include details of the expression tree at each point in the code generation process in the log file. Also defining -ftraceIR will output the graphs in the IR format.)
hqlfold
This is the constant folder. At the moment the only change is to ensure that fields that are assigned constants within the transform are processed correctly. Future work could add code to optimize quantile applied to an empty dataset, or selecting 1 division.
hqlmeta
Similar to the functions in hqlattr that calculate derived attributes, these functions are used to calculate how the rows coming out of an activity are sorted, grouped and distributed. It is vital to only preserve information that is guaranteed to be true - otherwise invalid optimizations might be performed on the rest of the expression tree.
reservedwords.cpp
A new entry indicating which category the keyword belongs to.
Finally we have the changes to the parser to recognise the new syntax:
hqllex.l
This file contains the lexer that breaks the ecl file into tokens. There are two new tokens - QUANTILE and SCORE.
Hqlgram.y
This file contains the grammar that matches the language. There are two productions - one that matches the version of QUANTILE with a transform and one without. (Two productions are used instead of an optional transform to avoid shift/reduce errors.)
hqlgram2.cpp
This contains the bulk of the code that is executed by the productions in the grammar. Changes here include new entries added to a case statement to get the text for the new tokens, and a new entry in the simplify() call. This helps reduce the number of valid tokens that could follow when reporting a syntax error.
Looking back over those changes, one reflection is that there are lots of different places that need to be changed. How does a programmer know which functions need to change, and what happens if some are missed? In this example, the locations were found by searching for an activity with a similar syntax e.g., no_soapcall_ds or no_normalize.
It is too easy to miss something, especially for somebody new to the code - although if you do then you will trigger a runtime internal error. It would be much better if the code was refactored so that the bulk of the changes were in one place. (See JIRA https://track.hpccsystems.com/browse/HPCC-13434 that has been added to track improvement of the situation.)
With these changes implemented the examples from the previous pull request now syntax check. The next stage in the process involves thinking through the details of how the activity will be implemented.
The next stage in adding a new activity to the system is to define the interface between the generated code and the engines. The important file for this stage is rtl/include/eclhelper.hpp, which contains the interfaces between the engines and the generated code. These interfaces define the information required by the engines to customize each of the different activities. The changes that define the interface for quantile are found in commit https://github.com/ghalliday/HPCC-Platform/commit/06534d8e9962637fe9a5188d1cc4ab32c3925010.
Adding a quantile activity involves the following changes:
ThorActivityKind - TAKquantile
Each activity that the engines support has an entry in this enumeration. This value is stored in the graph as the _kind attribute of the node.
ActivityInterfaceEnum - TAIquantilearg_1
This enumeration in combination with the selectInterface() member of IHThorArg provides a mechanism for helper interfaces to be extended while preserving backwards compatibility with older workunits. The mechanism is rarely used (but valuable when it is), and adding a new activity only requires a single new entry.
IHThorArg
This is the base interface that all activity interfaces are derived from. This interface does not need to change, but it is worth noting because each activity defines a specialized version of it. The names of the specialised interfaces follow a pattern; in this case the new interface is IHThorQuantileArg.
IHThorQuantileArg
The following is an outline of the new member functions, with comments on their use:
getFlags()
Many of the interfaces have a getFlags() function. It provides a concise way of returning several Boolean options in a single call - provided those options do not change during the execution of the activity. The flags are normally defined with explicit values in an enumeration before the interface. The labels often follow the pattern T<First-letter-of-activity>F<lowercase-name>, i.e. TQFxxx ~= Thor-Quantile-Flag-XXX.
getNumDivisions()
Returns how many parts to split the dataset into.
getSkew()
Corresponds to the SKEW() attribute.
queryCompare()
Returns an implementation of the interface used to compare two rows.
createDefault(rowBuilder)
A function used to create a default row - used if there are no input rows.
transform(rowBuilder, _left, _counter)
The function to create the output record from the input record and the partition number (passed as counter).
getScore(_left)
What weighting should be given to this row?
getRange(isAll, tlen, tgt)
Corresponds to the RANGE attribute.
Note that the different engines all use the same specialised interface - it contains a superset of the functions required by the different targets. Occasionally some of the engines do not need to use some of the functions (e.g., to serialize information between nodes) so the code generator may output empty implementations.
For each interface defined in eclhelper.hpp there is a base implementation class defined in eclhelper_base.hpp. The classes generated for each activity in a query by the code generator are derived from one of these base classes. Therefore we need to create a corresponding new class CThorQuantileArg. It often provides default implementations for some of the helper functions to help reduce the size of the generated code (e.g., getScore returning 1).
Often the process of designing the helper interface is dynamic. As the implementation is created, new options or possibilities for optimizations appear. These require extensions and changes to the helper interface in order to be implemented by the engines. Once the initial interface has been agreed, work on the code generator and the engines can proceeded in parallel. (It is equally possible to design this interface before any work on the parser begins, allowing more work to overlap.)
There are some more details on the contents of thorhelper.hpp in the documentation ecl/eclcc/WORKUNIT.rst within the HPCC repository.
Adding a new activity to the code generator is (surprisingly!) a relatively simple operation. The process is more complicated if the activity also requires an implementation that generates inline C++, but that only applies to a small subset of very simple activities, e.g., filter, aggregate. Changes to the code generator also tend to be more substantial if you add a new type, but that is also not the case for the quantile activity.
For quantile, the only change required is to add a function that generates an implementation of the helper class. The code for all the different activities follows a very similar pattern - generate input activities, generate the helper for this activity, and link the input activities to this new activity. It is often easiest to copy the boiler-plate code from a similar activity (e.g., sort) and then adapt it. (Yes, some of this code could also be refactored... any volunteers?) There are a large number of helper functions available to help generate transforms and other member functions, which also simplifies the process.
Most of the code should be self explanatory, but one item is worth highlighting. The code generator builds up a structure in memory that represents the C++ code that is being generated. The BuildCtx class is used to represent a location within that generated code where new code can be inserted. The instance variable contains several BuildCtx members that are used to represent locations to generate code within the helper class (classctx, nestedctx, createctx and startctx). They are used for different purposes:
classctx
Used to generate any member functions that can be called as soon as the helper object has been created, e.g., getFlags().
nestedctx
Used to generate nested member classes and objects - e.g., comparison classes.
startctx
Any function that may return a value that depends on the context/parent activity. For example if QUANTILE is used inside the TRANSFORM of a PROJECT, the number of partition points may depend on a field in the LEFT row of the PROJECT. Therefore the getNumDivisions() member function needs to be generated inside instance->startctx. These functions can only be called by the engine after onCreate() and onStart() have been called to set up the current context.
createctx
Really, this is a historical artefact from many years ago. It was originally used for functions that could be dependent on a global expression, but not a parent row. Almost all such restrictions have since been removed, and those that remain should probably be replaced with either classctx or startctx.
The only other change is to extend the switch statement in common/thorcommon/thorcommon.cpp to add a text description of the activity.
With the code generator outputting all the information we need, we can now implement the activity in one of the engines. (As I mentioned previously, in practice this is often done in parallel with adding it to the code generator.) Roxie and hThor are the best engines to start with because most of their activities run on a single node - so the implementations tend to be less complicated. It is also relatively easy to debug them, by compiling to create a stand-alone executable, and then running that executable inside a debugger. The following description walks-through the roxie changes:
Code is added to ccdquery to process the new TAKquantile activity kind, and create a factory object of the correct type. The implementation of the factory class is relatively simple - it primarily contains a method for creating an instance of the activity class. Some factories create instances of the helper and cache any information that never changes (in this case the value returned by getFlags(), which is a very marginal optimization).
The classes that implement the existing sort algorithms are extended to return the sorted array in a single call. This allows the quicksort variants to be implemented more efficiently.
The class CRoxieServerQuantileActivity contains the code for implementing the quantile activity. It has the following methods:
Constructor
Extracts any information from the helper that does not vary, and initializes all member variables.
start()
This function is called before the graph is executed. It evaluates any helper methods that might vary from execution to execution (e.g., getRange(), numDivisions()), but which do not depend on the current row.
reset()
Called when a graph has finished executing - after an activity has finished processing all its records. It is used to clean up any variables, and restore the activity ready for processing again (e.g., if it is inside a child query).
needsAllocator()
Returns true if this activity creates new rows.
nextInGroup()
The main function in the activity. This function is called by whichever activity is next in the graph to request a new row from the quantile activity. The functions should be designed so they return the next row as quickly as possible, and delay any processing until it is needed. In this case the input is not read and sorted until the first row is requested.
Note, the call to the helper.transform() returns the size of the resulting row, and returns zero if the row should be skipped. The call to finaliseRowClear() after a successful row creation is there to indicate that the row can no longer be modified, and ensures that any child rows will be correctly freed when the row is freed.
The function also contains extra logic to ensure that groups are implemented correctly. The end of a group is marked by returning a single NULL row, the end of the dataset by two contiguous NULL rows. It is important to ensure that a group that has all its output rows skipped doesn't return two NULLs in a row - hence the checks for anyThisGroup.
With those changes in place, the second commit https://github.com/ghalliday/HPCC-Platform/commit/aeaa209092ea1af9660c6908062c1b0b9acff36b adds support for the RANGE, FIRST, and LAST attributes. It also optimizes the cases where the input is already sorted, and the version of QUANTILE which does not include a transform. (If you are looking at the change in github then it is useful to ignore whitespace changes by appending ?w=1 to the URL). The main changes are
Extra helper methods called in start() to obtain the range.
Optimize the situation where the input is known to be sorted by reading the input rows directly into the "sorted" array.
Extra checks to see if this quantile should be included in the output (FIRST,LAST,RANGE,DEDUP)
An optimization to link the incoming row if the transform does not modify it, by testing the TQFneedtransform flag.
Because I could. Many of the pieces needed for Roxie were already created for use in other systems – ECL language, code generator, index creation in Thor, etc. Indexes could be used by Moxie, but that relied on monolithic single-part indexes, was single-threaded (forked a process per query) and had limited ability to do any queries beyond simple index lookups. ECL had already proved itself as a way to express more complex queries concisely, and the concept of doing the processing next to the data had been proved in hOle and Thor, so Roxie – using the same concept for online queries using indexes – was a natural extension of that, reusing the existing index creation and code generation, but adding a new run-time engine geared towards pre-deployed queries and sending index lookup requests to the node holding the index data.
The code generator creates a graph (DAG) representing the query, with one node per activity and links representing the inputs and dependencies. There is also a helper class for each activity.
Roxie loads this graph for all published queries, creating a factory for each activity and recording how they are linked. When a query is executed, the factories create the activity instances and link them together. All activities without output activities (known as ‘sinks’) are then executed (often on parallel threads), and will typically result in a value being written to a workunit, to the socket that the query was received in, or to a global “context” area where subsequent parts of the query might read it.
Data is pulled through the activity graph, by any activity that wants a row from its input requesting it. Evaluation is therefore lazy, with data only calculated as needed. However, to reduce latency in some cases activities will prepare results ahead of when they are requested – for example an index read activity will send the request to the agent(s) as soon as it is started rather than waiting for the data to be requested by its downstream activity. This may result in wasted work, and in some cases may result in data coming back from an agent after the requesting query has completed after discovering it didn’t need it after all – this results in the dreaded “NO msg collator found – using default” tracing (not an error but may be indicative of a query that could use some tuning).
Before requesting rows from an input, it should be started, and when no more rows are required it should be stopped. It should be reset before destruction or reuse (for example for the next row in a child query).
Balancing the desire to reduce latency with the desire to avoid wasted work can be tricky. Conditional activities (IF etc) will not start their unused inputs, so that queries can be written that do different index reads depending on the input. There is also the concept of a “delayed start” activity – I would need to look at the code to remind myself of how those are used.
Splitter activities are a bit painful – they may result in arbitrary buffering of the data consumed by one output until another output is ready to request a row. It’s particularly complex when some of the outputs don’t start at all – the splitter needs to keep track of how many of the inputs have been started and stopped (an input that is not going to be used must be stopped, so that splitters know not to keep data for them). Tracking these start/stop/reset calls accurately is very important otherwise you can end up with weird bugs including potential crashes when activities are destroyed. Therefore we report errors if the counts don’t tally properly at the end of a query – but working out where a call was missed is often not trivial. Usually it’s because of an exception thrown from an unexpected place, e.g. midway through starting.
Note that row requests from the activities above a splitter may execute on whichever thread downstream from the splitter happens to need that particular row first.
The splitter code for tracking whether any of the downstream activities still need a row is a bit hairy/inefficient, IIRC. There may be scope to optimize (but I would recommend adding some good unit test cases first!)
When there are multiple agents fulfilling data on a channel, work is shared among them via a hash of the packet header, which is used to determine which agent should work on that packet. However, if it doesn’t start working on it within a short period (either because the node is down, or because it is too busy on other in-flight requests), then another node may take over. The IBYTI messages are used to indicate that a node has started to work on a packet and therefore there is no need for a secondary to take over.
The priority of agents as determined by the packet hash is also used to determine how to proceed if an IBYTI is received after starting to work on a request. If the IBYTI is from a lower priority buddy (sub-channel) then it is ignored, if it’s from a higher priority one then the processing will be abandoned.
When multicast is enabled, the IBYTI is sent on the same multicast channel as the original packet (and care is needed to ignore ones sent by yourself). Otherwise it is sent to all buddy IPs.
Nodes keep track of how often they have had to step in for a supposedly higher priority node, and reduce their wait time before stepping in each time this happens, so if a node has crashed then the buddy nodes will end up taking over without every packet being delayed.
(QUESTION – does this result in the load on the first node after the failed node getting double the load?)
Newer code for cloud systems (where the topology may change dynamically) send the information about the buddy nodes in the packet header rather than assuming all nodes already have a consistent version of that information. This ensures that all agents are using the same assumptions about buddy nodes and their ordering.
An index is basically a big sorted table of the keyed fields, divided into pages, with an index of the last row from each page used to be able to locate pages quickly. The bottom level pages (‘leaves’) may also contain payload fields that do not form part of the lookup but can be returned with it.
Typical usage within LN Risk tends to lean towards one of two cases:
Many keyed fields with a single “ID” field in the payload
A single “ID” field in the key with many “PII” fields in the payload.
There may be some other cases of note too though – e.g. an error code lookup file which heavily, used, or Boolean search logic keys using smart-stepping to implement boolean search conditions.
It is necessary to store the index pages on disk compressed – they are very compressible – but decompression can be expensive. For this reason traditionally we have maintained a cache of decompressed pages in addition to the cache of compressed pages that can be found in the Linux page cache. However, it would be much preferred if we could avoid decompressing as much as possible, ideally to the point where no significant cache of the decompressed pages was needed.
Presently we need to decompress to search, so we’ve been looking at options to compress the pages in such a way that searching can be done using the compressed form. The current design being played with here uses a form of DFA to perform searching/matching on the keyed fields – the DFA data is a compact representation of the data in the keyed fields but is also efficient to use as for searching. For the payload part, we are looking at several options (potentially using more than one of them depending on the exact data) including:
Do not compress (may be appropriate for ID case, for example)
Compress individual rows, using a shared dictionary (perhaps trained on first n rows of the index)
Compress blocks of rows (in particular, rows that have the same key value)
A fast (to decompress) compression algorithm that handles small blocks of data efficiently is needed. Zstd may be one possible candidate.
Preliminary work to enable the above changes involved some code restructuring to make it possible to plug in different compression formats more easily, and to vary the compression format per page.
It’s used in the cloud to ensure that all nodes can know the IP addresses of all agents currently processing requests for a given channel. These addresses can change over time due to pod restarts or scaling events. Nodes report to the topology server periodically, and it responds to them with the current topology state. There may be multiple topology servers running (for redundancy purposes). If so all reports should go to all, and it should not matter which one’s answer is used. (QUESTION – how is the send to all done?)
All IFileIO objects used to read files from Roxie are instantiated as IRoxieLazyFileIO objects, which means:
The underlying file handles can be closed in the background, in order to handle the case where file handles are a limited resource. The maximum (and minimum) number of open files can be configured separately for local versus remote files (sometimes remote connections are a scarcer resource than local, if there are limits at the remote end).
The actual file connected to can be switched out in the background, to handle the case where a file read from a remote location becomes unavailable, and to switch to reading from a local location after a background file copy operation completes.
Original IBYTI implementation allocated a thread (from the pool) to each incoming query packet, but some will block for a period to allow an IBYTI to arrive to avoid unnecessary work. It was done this way for historical reasons - mainly that the addition of the delay was after the initial IBYTI implementation, so that in the very earliest versions there was no priority given to any particular subchannel and all would start processing at the same time if they had capacity to do so.
This implementation does not seem particularly smart - in particular it's typing up worker threads even though they are not actually working, and may result in the throughput of the Roxie agent being reduced. For that reason an alternative implementation (controlled by the NEW_IBYTI flag) was created during the cloud transition which tracks what incoming packets are waiting for IBYTI expiry via a separate queue, and they are only allocated to a worker thread once the IBYTI delay times out.
So far the NEW_IBYTI flag has only been set on containerized systems (simply to avoid rocking the boat on the bare-metal systems), but we may turn on in bare metal too going forward (and if so, the old version of the code can be removed sooner or later).
Sometimes when developing/debugging Roxie features, it's simplest to run a standalone executable. Using server mode may be useful if wanting to debug server/agent traffic messaging.
For example, to test IBYTI behaviour on a single node, use
Roxie (optionally) maintains a list of the most recently accessed file pages (in a circular buffer), and flushes this information periodically to text files that will persist from one run of Roxie to the next. On startup, these files are processed and the relevant pages preloaded into the linux page cache to ensure that the "hot" pages are already available and maximum performance is available immediately once the Roxie is brought online, rather than requiring a replay of a "typical" query set to heat the cache as used to be done. In particular this should allow a node to be "warm" before being added to the cluster when autoscaling.
There are some questions outstanding about how this operates that may require empirical testing to answer. Firstly, how does this interact with volumes mounted via k8s pvc's, and in particular with cloud billing systems that charge per read. Will the reads that are done to warm the cache be done in large chunks, or will they happen one linux page at a time? The code at the Roxie level operates by memory-mapping the file then touching a byte within each linux page that we want to be "warm", but does the linux paging subsystem fetch larger blocks? Do huge pages play a part here?
Secondly, the prewarm is actually done by a child process (ccdcache), but the parent process is blocked while it happens. It would probably make sense to at allow at least some of the other startup operations of the parent process to proceed in parallel. There are two reasons why the cache prewarm is done using a child process. Firstly is to allow there to be a standalone way to prewarm prior to launching a Roxie, which might be useful for automation in some bare-metal systems. Secondly, because there is a possibility of segfaults resulting from the prewarm if the file has changed size since the cache warming was done, it is easier to contain, capture, and recover from such faults in a child process than it would be inside Roxie. However, it would probably be possible to avoid these segfaults (by checking more carefully against file size before trying to warm a page, for example) and then link the code into Roxie while still keeping the code common with the standalone executable version.
Thirdly, need to check that the prewarm is complete before adding a new agent to the topology. This is especially relevant if we make any change to do the prewarm asynchronously.
Fourthly, there are potential race conditions when reading/writing the file containing cache information, since this file may be written by any agent operating on the same channel, at any time.
Fifthly, how is the amount of information tracked decided? It should be at least related to the amount of memory available to the linux page cache, but that's not a completely trivial thing to calculate. Should we restrict to the most recent N when outputting, where N is calculated from, for example /proc/meminfo's Active(file) value? Unfortunately on containerized sytems that reflects the host, but perhaps /sys/fs/cgroup/memory.stat can be used instead?
When deciding how much to track, we can pick an upper limit from the pod's memory limit. This could be read from /sys/fs/cgroup/memory.max though we currently read from the config file instead. We should probably (a) subtract the roxiemem size from that and (b) think about a value that will work on bare-metal and fusion too. However, because we don't dedup the entries in the circular buffer used for tracking hot pages until the info is flushed, the appropriate size is not really the same as the memory size.
We track all reads by page, and before writing also add all pages in the jhtree cache with info about the node type. Note that a hit in the jhtree page cache won't be noted as a read OTHER than via this last-minute add.
This isn't really specific to Roxie, but was originally added for federated Roxie systems...
When a Roxie query (or hthor/thor) makes a SOAPCALL, there is an option to specify a list of target gateway IPs, and failover to the next in the list if the first does not respond in a timely fashion. In order to avoid this "timely fashion" check adding an overhead to every query made when a listed gateway is unavailable, we maintain a "blacklist" of nodes that have been seen to fail, and do not attempt to connect to them. There is a "deblacklister" thread that checks periodically whether it is now possible to connect to a previously-blacklisted gateway, and removes it from the list if so.
There are a number of potential questions and issues with this code:
It would appear that a blacklist is applied even when there is only one gateway listed. In this case, the blacklist may be doing more harm than good? I'm not sure that is true - it is still causing rapid failures in cases that are never going to work...
Even when there is only one gateway listed, a blacklist MIGHT still be useful (you don't really want EVERY query to block trying to connect, if the gateway is down - may prefer a fast failure). Also applies when there are multiple records, all being passed to a gateway, and with an ONFAIL.
Is the blacklist shared between all queries? I'm pretty sure it is NOT shared across Roxie nodes... Looks like it is a global object, shared between all queries and activities. However, connections that were started in parallel will all be reported as failed rather than blacklisted, which can make it look like it is maintained per-activity.
Is it only a failed connect that leads to blacklisting, or does a slow/error response also cause a gateway endpoint to be blacklisted? It's only a failed connect.
When deblacklisting, can we check any condition other than "Successfully connected"? If not, blacklisting for any reason other than "Did not connect" feels like a recipe for problems. We only check for a connection (and correspondingly only blacklist for a failed connection).
Are we ever using the functionality where there are more than one gateway listed? Most of the time a load-balancer is a preferable solution...
The "deblacklister" thread seems to add an escalating delay between attempts. Is this delay ever reset? Is it configurable? Is it appropriate?
There's a thread (in the blacklister's pool) for each blacklisted endpoint. These threads will never go away if the endpoint does not recover...
There's a delay of up to 10 seconds in terminating caused by the deblacklister's connect having to timeout before it notices that we are stopping. Can we close the socket as well as interrupting the semaphore?
Does the "reconnect" attempt from the deblacklister cause any pain for the server it is connecting to? Lots of connect attempts without any data could look like a DoS attack...
Retries/timeout seems to translate to Owned<ISocketConnectWait> scw = nonBlockingConnect(ep, timeoutMS == WAIT_FOREVER ? 60000 : timeoutMS*(retries+1)); I am not sure that is correct (a single attempt to connect with a long timeout doesn't feel like it is the same as multiple attempts with shorter timeouts, for example if there is a load balancer in the mix).
Perhaps an option to not use blacklister would solve the immediate issue?
The blacklister uses an array of endpoints - If there were a lot blacklisted, a hash table would be better
Hints to control behaviour of deblacklister would behave unpredictably if multiple activities connected to the same endpoint with different hints unless we make the blacklist lookup match the hint values too.
Deblacklister should use nonBlockingConnect too.
Should the scope of the blacklist be different? Possible scopes are:
Shared across all queries/activities (current behaviour)
Specific to an activity, but shared across queries (i.e. owned by the activity factory)
Specific to all activities in a deployed query (i.e. owned by the query factory)
Specific to a particular activity instance (i.e. owned by the activity object)
Specific to a particular query instance (i.e. owned by the query object)
Options 2 and 4 above would allow all aspects of the blacklisting behaviour to be specified by options on the SOAPCALL. We could control whether or not the blacklister is to be used at all via a SOAPCALL option with any of the above...
The HPCC Platform includes a rudimentary performance tracing feature using periodic stack capture to generate flame graphs. Roxie supports this in 3 ways:
If expert/@profileStartup is set in roxie config, a flame graph is generated for operations during Roxie startup phase.
If @perf is set on an incoming query, a flame graph is generated for the lifetime of that query's execution, and returned along with the query results
If expert/perftrace is set in roxie config, one-shot roxie queries (e.g. eclagent mode) generate a flame graph (currently just to a text file).
The perf trace operates as follows:
A child process is launched that runs the doperf script. This samples the current stack(s) every 0.2s (configurable) to a series of text files.
When tracing is done, these text files are "folded" via a perl script that notes every unique stack and how many times it was seen, one line per unique stack
This folded stack list is filtered to suppress some stacks that are not very interesting
The filtered folded stack list is passed to another perl script that generates an svg file.
The basic info captured at step 1 (or maybe 2) could also be analysed to give other insights, such as:
A list of "time in function, time in children of function".
An expanded list of callers to __GI___lll_lock_wait and __GI___lll_lock_wake, to help spot contended critsecs.
Unfortunately some info present in the original stack text files is lost in the folded summary - in particular related to the TID that the stack is on. Can we spot lifetimes of threads and/or should we treat "stacks" on different threads as different? Thread pools might render this difficult though. There is an option in stack-collapse-elfutils.pl to include the TID when considering whether stacks match, so perhaps we should just (optionally) use that.
In localAgent mode, the global queueManager object (normally a RoxieUdpSocketQueueManager) is replaced by a RoxieLocalQueueManager. Outbound packets are added directly to target queue, inbound are packed into DataBuffers.
There is also "local optimizations" mode where any index operation reading a one-part file (does the same apply to one-part disk files?) just reads it directly on the server (regardless of localAgent setting). Typically still injected into receiver code though as otherwise handling exception cases, limits etc would all be duplicated/messy. Rows created in localOptimization mode are created directly in the caller's row manager, and are injected in serialized format.
Why are inbound not created directly in the desired destination's allocator and then marked as serialized? Some lifespan issues... are they insurmountable? We do pack into dataBuffers rather than MemoryBuffers, which avoids a need to copy the data before the receiver can use it. Large rows get split and will require copying again, but we could set dataBufferSize to be bigger in localAgent mode to mitigate this somewhat.
What is the lifespan issue? In-flight queries may be abandoned when a server-side query fails, times out, or no longer needs the data. Using DataBuffer does not have this issue as they are attached to the query's memory manager/allocation once read. Or we could bypass the agent queue altogether, but rather more refactoring needed for that (might almost be easier to extent the "local optimization" mode to use multiple threads at that point)
abortPending, replyPending, and abortPendingData methods are unimplemented, which may lead to some inefficiencies?
Requests from server to agents are send via UDP (and have a size limit of 64k as a result). Historically they were sent using multicast to go to all agents on a channel at the same time, but since most cloud providers do not support multicast, there has long been an option to avoid multicast and send explicitly to the agent IPs. In bare metal systems these IPs are known via the topology file, and do not change. In cloud systems the topology server provides the IPs of all agents for a channel.
In cloud systems, the list of IPs that a message was sent to is included in the message header, so that the IBYTI messages can be sent without requiring that all agents/servers have the same topology information at any given moment (they will stay in sync because of topology server, but may be temporarily out of sync when nodes are added/removed, until next time topology info is retrieved). This is controled by the SUBCHANNELS_IN_HEADER define.
Packets back from agents to server go via the udplib message-passing code. This can best be described by looking at the sending and receiving sides separately.
When sending, results are split into individual packets (DataBuffers), each designed to be under 1 MTU in size. Traditionally this meant they were 1k, but they can be set larger (8k is good). They do have to be a power of 2 because of how they are allocated from the roxiemem heap. The sender maintains a set of UdpReceiverEntry objects, one for each server that it is conversing with. Each UdpReceiverEntry maintains multiple queues of data packets waiting to be sent, one queue for each priority. The UdpReceiverEntry maintains a count of how many packets are contained across all its queues in packetsQueued, so that it knows if there is data to send.
The priority levels are: 0: Out Of Band 1: Fast lane 2: Standard
This is designed to allow control information to be sent without getting blocked by data, and high priority queries to avoid being blocked by data going to lower priority ones. The mechanism for deciding what packet to send next is a little odd though - rather than sending all higher-priorty packets before any lower-priority ones, it round robins across the queues sending up to N^2 from queue 0 then up to N from queue 1 then 1 from queue 2, where N is set by the UdpOutQsPriority option, or 1 if not set. This may be a mistake - probably any from queue 0 should be sent first, before round-robining the other queues in this fashion.
UdpReceiverEntry objects are also responsible for maintaining a list of packets that have been sent but receiver has not yet indicated that they have arrived.
If an agent has data ready for a given receiver, it will send a requestToSend to that receiver, and wait for a permitToSend response. Sequence numbers are used to handle situations where these messages get lost. A permitToSend that does not contain the expected sequence number is ignored.
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/userdoc/AzureTipsTricks.html b/devdoc/userdoc/AzureTipsTricks.html
new file mode 100644
index 00000000000..f01b314e579
--- /dev/null
+++ b/devdoc/userdoc/AzureTipsTricks.html
@@ -0,0 +1,24 @@
+
+
+
+
+
+ HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Documentation in this directory is targeted to end-users of the HPCC Systems Platform. This is less-formal documentation intended to be produced and released more quickly than the published HPCC documentation. See HPCC documentation if you would like to contribute to our official docs.
1. go to the resource group service page (https://portal.azure.com/#view/HubsExtension/BrowseResourceGroups)
+2. "Add filter"
+3. Filter on "Admin"
+4. Set the Value to "ALL" or select the names you are interested in (the names in the list are the only ones that have the Admin tag set) This works for all other fields that you can filter on
How do I create a dashboard in Azure?
Answer:
Login to the Azure portal.
+
+Click on the `hamburger icon`, located at the top left corner of the page.
+
+Click on `Dashboard` to go to your dashboards.
+
+Click on `Create`, located at the top left corner.
+
+Click on the `Custom` tile.
+
+Edit the input box to name your dashboard.
+
+Click on `Resource groups` in the tile gallery.
+
+Click `Add`.
+
+Click and drag the `lower right corner` of the tile to resize it to your likings.
+
+Click `Save` to save your settings.
+
+You should now be taken to your new dashboard.
+
+Click on your new dashboard tile.
+
+Click on `Add filter`, located at the top center of the page.
+
+Click on the `Filter` input box to reveal the tags.
+
+Select the `Admin` tag.
+
+Click on the `Value` input box.
+
+Click on `Select all` to unselect all.
+
+Select your name.
+
+Click on `Apply`.
+
+Next, click on `Manage view`, located at the top left of the page.
+
+Select `Save view`.
+
+Enter a name for the view in the input box.
+
+Click `Save`
+
+
+
+
+
+Click on `Manage View`, located at the top left of the page
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/userdoc/copilot/CopilotPromptTips.html b/devdoc/userdoc/copilot/CopilotPromptTips.html
new file mode 100644
index 00000000000..fe0b6c55b08
--- /dev/null
+++ b/devdoc/userdoc/copilot/CopilotPromptTips.html
@@ -0,0 +1,24 @@
+
+
+
+
+
+ Copilot Prompt Tips | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Unlock the full potential of GitHub Copilot with these carefully curated prompts designed to streamline your workflow and enhance productivity.
Whether you're summarizing complex texts, brainstorming innovative ideas, or creating detailed guides, these prompts will help you get the most out of Copilot's capabilities.
Dive in and discover how these simple yet powerful prompts can save you time and effort in your daily tasks.
These prompts are more focused and are meant to accomplish a specific purpose. They are included here to spark your imagination.
If you think of others that could be included here to share with the world, please send your ideas to docfeedback@hpccsystems.com.
Write comments for this ECL code in javadoc format
Create an ECL File Definition, record structure, and inline dataset with [N (number)] of records with the following fields [list of fields]
Example: Create an ECL file definition with a record structure and an inline dataset with 20 records for a dataset of animals with the following fields: ReferenceNumber, AnimalName, ClassName, FamilyName, Color, Size, and LifeExpectancy.
Write ECL code to classify records into [categories].
Example: Write ECL code to classify customer feedback into neutral, negative, or positive categories.
How to Avoid AI Hallucinations with Good Prompts
AI hallucinations refer to instances where artificial intelligence systems generate information or responses that are incorrect, misleading, or entirely fabricated.
Hallucinations can occur due to various reasons, such as limitations in the training data, inherent biases, or the AI's attempt to provide an answer even when it lacks sufficient context or knowledge.
Understanding and mitigating AI hallucinations is crucial for ensuring the reliability and accuracy of AI-driven applications.
Creating effective prompts is essential to minimize AI hallucinations. Here are some tips to help you craft prompts that lead to accurate and reliable responses:
Be Specific and Clear: Ambiguous prompts can lead to incorrect or irrelevant answers. Clearly define the task and provide specific instructions.
Example: Instead of asking "What is AI?", ask "Explain the concept of artificial intelligence and its primary applications in healthcare."
Provide Context: Give the AI enough background information to understand the task. This helps in generating more accurate responses.
Example: "Given the following text about climate change, summarize the key points in a bullet list."
Use Constraints: Limit the scope of the response by specifying constraints such as word count, format, or specific details to include.
Example: "List 5 benefits of renewable energy in a bullet list, each point not exceeding 20 words."
Ask for Evidence or Sources: Encourage the AI to provide evidence or cite sources for the information it generates.
Example: "Explain the impact of social media on mental health and provide references to recent studies."
Iterative Refinement: Start with a broad prompt and refine it based on the initial responses to get more accurate results.
Example: Begin with "Describe the process of photosynthesis." If the response is too vague, refine it to "Describe the process of photosynthesis in plants, including the role of chlorophyll and sunlight."
By following these guidelines, you can reduce the likelihood of AI hallucinations and ensure that the responses generated are accurate and useful.
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/userdoc/roxie/FAQ.html b/devdoc/userdoc/roxie/FAQ.html
new file mode 100644
index 00000000000..a16cf49905f
--- /dev/null
+++ b/devdoc/userdoc/roxie/FAQ.html
@@ -0,0 +1,47 @@
+
+
+
+
+
+ ROXIE FAQs | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
How I can compile a query on a containerized or cloud-based system?
Answer:
Same way as bare metal. Command line, or with the IDE, or from ECL Watch. Just point to the HPCC Systems instance to compile.
+For Example:
+ecl deploy <target> <file>
How do I copy queries from an on-prem cluster to Azure?
Answer:
The copy query command – use the Azure host name or IP address for the target.
+For example:
+ecl queries copy <source_query_path> <target_queryset>
How can I get the IP address for the Azure target cluster?
Answer:
Use the "kubectl get svc" command. Use the external IP address listed for ECL Watch.
+kubectl get svc
Do we have to have use the DNSName or do we need to use the IP address?
Answer:
If you can reach ECL Watch with the DNS Name then it should also work for the command line.
How can I find the ECL Watch or Dali hostname?
Answer:
If you did not set up the containerized instance, then you need to ask your Systems Administrator or whomever set it up..
How do I publish a package file?
Answer:
Same way as bare metal.
+To add a new package file: ecl packagemap add or
+To copy exisitng package file : ecl packagemap copy
How do I check the logs?
Answer:
kubectl log <podname>
+in addition you can use -f (follow) option to tail the logs. Optionally you can also issue the <namespace> parameter.
+For example:
+kbectl log roxie-agent-1-3b12a587b –namespace MyNameSpace
+Optionally, you may have implemented a log-processing solution such as the Elastic Stack (elastic4hpcclogs).
How do I get the data on to Azure?
Answer:
Use the copy query command and copy or add the Packagemap.
+With data copy start in the logs…copy from remote location specified if data doesn’t exist on the local system.
+The remote location is the remote Dali (use the --daliip=<daliIP> parameter to specify the remote Dali)
+You can also use ECL Watch.
How can I start a cloud cluster? (akin to the old Virtual Box image)?
Answer:
Can use Docker Desktop, or Azure or any cloud provider and install the HPCC Systems Cloud native helm
+charts
How can I show the ECL queries that are published to a given Roxie?
Answer:
Can use WUListQueries
+For example:
+https://[eclwatch]:18010/WsWorkunits/WUListQueries.json?ver_=1.86&ClusterName=roxie&CheckAllNodes=0
I set up persistent storage on my containerized HPCC Systems, and now it won't start. Why?
Answer:
One possible reason may be that all of the required storage directories are not present. The directories for ~/
+hpccdata/dalistorage, hpcc-data, debug, queries, sasha, and dropzone are all required to exist or your cluster may not start.
Are there any new methods available to work with queries?
Answer:
Yes. There is a new method available ServiceQuery.
+https://[eclwatch]:18010/WsResources/ServiceQuery?ver_=1.01&
+For example Roxie Queries:
+https://[eclwatch]:18010/WsResources/ServiceQuery?ver_=1.01&Type=roxie
+or WsECL (eclqueries)
+https://[eclwatch]:18010/WsResources/ServiceQuery?ver_=1.01&Type=eclqueries
+
+
+
+
\ No newline at end of file
diff --git a/devdoc/userdoc/troubleshoot/ClientsToolIssues.html b/devdoc/userdoc/troubleshoot/ClientsToolIssues.html
new file mode 100644
index 00000000000..ab238d95dcc
--- /dev/null
+++ b/devdoc/userdoc/troubleshoot/ClientsToolIssues.html
@@ -0,0 +1,24 @@
+
+
+
+
+
+ How to resolve issues installing the ECL IDE / Client tools | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
Some users may experience an issue when trying to install the ECL IDE / client tools version 8.10 and later. If you get an error saying something like, Windows Defender SmartScreen prevented access and there is no option other than "Don't Run":
Go to the folder where you downloaded the file.
Once there go to the app file properties click unblock
The ecl-bundle executable (normally executed from the ecl executable by specifying 'ecl bundle XXX' is designed to manipulate ecl bundle files
these are self-contained parcels of ECL code, packaged into a compressed file like a zip or .tar.gz file, that can be downloaded, installed, removed etc from an ECL installation.
The metadata for ECL bundles is described using an exported module called Bundle within the bundle's source tree - typically, this means that a file called Bundle.ecl will be added to the highest level of the bundle's directory tree. In order to extract the information from the Bundle module, eclcc is run in 'evaluate' mode (using the -Me option) which will parse the bundle module and output the required fields to stdout.
ecl-bundle also executes eclcc (using the --showpaths option) to determine where bundle files are to be located.
In order to make versioning easier, bundle files are not copied directly into the bundles directory. A bundle called "MyBundle" that announces itself as version "x.y.z" will be installed to the directory
$BUNDLEDIR/_versions/MyBundle/x.y.z
A "redirect" file called MyBundle.ecl is then created in $BUNDLEDIR, which redirects any IMPORT MyBundle statement to actually import the currently active version of the bundle in _versions/MyBundle/x.y.z
By rewriting this redirect file, it is possible to switch to using a different version of a bundle without having to uninstall and reinstall.
In a future release, we hope to make it possible to specify that bundle A requires version X of bundle B, while bundle C requires version Y of bundle B. That will require the redirect files to be 'local' to a bundle (and will require that bundle B uses a redirect file to ensure it picks up the local copy of B when making internal calls).
An IBundleInfo represents a specific copy of a bundle, and is created by explicitly parsing a snippet of ECL that imports it, with the ECL include path set to include only the specified bundle.
An IBundleInfoSet represents all the installed versions of a particular named bundle.
An IBundleCollection represents all the bundles on the system.
Every individual subcommand is represented by a class derived (directly or indirectly) from EclCmdCommon. These classes are responsible for command-line parsing, usage text output, and (most importantly) execution of the desired outcomes.
+
+
+
+
\ No newline at end of file
diff --git a/ecllibrary/StyleGuide.html b/ecllibrary/StyleGuide.html
new file mode 100644
index 00000000000..228e27d22e0
--- /dev/null
+++ b/ecllibrary/StyleGuide.html
@@ -0,0 +1,43 @@
+
+
+
+
+
+ ECL standard library style guide | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
my_record := RECORD
+ INTEGER4 id;
+ STRING firstname{MAXLENGTH(40)};
+ STRING lastname{MAXLENGTH(50)};
+END;
+
+/**
+ * Returns a dataset of people to treat with caution matching a particular lastname. The
+ * names are maintained in a global database of undesirables.
+ *
+ * @param search_lastname A lastname used as a filter
+ * @return The list of people
+ * @see NoFlyList
+ * @see MorePeopleToAvoid
+ */
+
+EXPORT DodgyCharacters(STRING search_lastname) := FUNCTION
+ raw_ds := DATASET(my_record, 'undesirables', THOR);
+ RETURN raw_ds(last_name = search_lastname);
+END;
Some additional rules for attributes in the library:
Services should be SHARED and EXPORTed via intermediate attributes
All attributes must have at least one matching test. If you're not on the test list you're not coming in.
+
+
+
+
\ No newline at end of file
diff --git a/hashmap.json b/hashmap.json
new file mode 100644
index 00000000000..60d74ce9d0f
--- /dev/null
+++ b/hashmap.json
@@ -0,0 +1 @@
+{"build_me.md":"Cs6Xx-2H","cmake_modules_documentation.md":"BLw9X_2f","devdoc_codegenerator.md":"DjbPmndU","devdoc_codereviews.md":"BB4WmPQp","devdoc_codesubmissions.md":"DoLTBfxy","devdoc_devdocs.md":"DbnKa3rJ","devdoc_development.md":"Cz70vUo4","devdoc_docs_contributedocs.md":"B6UEh3AF","devdoc_docs_doctemplate.md":"DO145SYO","devdoc_docs_hpccstyleguide.md":"ttTlrCG0","devdoc_gitauthenticate.md":"DWDVKYp3","devdoc_ldapsecuritymanager.md":"Cty5AMLN","devdoc_memorymanager.md":"lMxAbcSH","devdoc_metrics.md":"BVV7YlV-","devdoc_newactivity.md":"Cwh2eRu1","devdoc_newfileprocessing.md":"DGLPSf2v","devdoc_readme.md":"D15419X_","devdoc_roxie.md":"C6oOX-36","devdoc_securityconfig.md":"C2-pRa18","devdoc_securityuserauthentication.md":"DbBHZTGf","devdoc_styleguide.md":"6RHWelMX","devdoc_userbuildassets.md":"RFQFpFTM","devdoc_userdoc_azure_tipsandtricks.md":"2OyOG7Og","devdoc_userdoc_azuretipstricks.md":"BUnTMakG","devdoc_userdoc_blogs.md":"D5ZHWDBF","devdoc_userdoc_copilot_copilotprompttips.md":"_YhzUQ6a","devdoc_userdoc_readme.md":"6WX0TGeB","devdoc_userdoc_roxie_faq.md":"Cbz7RPiY","devdoc_userdoc_troubleshoot_clientstoolissues.md":"Buqpsc4y","devdoc_userdoc_wikiguidelines.md":"BcZTSYE9","devdoc_versionsupport.md":"DqUAdEd9","devdoc_workunits.md":"DJg1TI-D","ecl_ecl-bundle_documentation.md":"rUiAyrKM","ecllibrary_styleguide.md":"CjMQ9yWq","index.md":"C4lg4w25","readme.md":"aSZY6Tfx","system_httplib_readme.md":"WIkDaVQZ","system_masking_include_readme.md":"CNC1sarR","system_masking_plugins_datamasker_readme.md":"DKeIe1BA","system_security_plugins_jwtsecurity_readme.md":"B1LGJUHj","testing_regress_cleanupreadme.md":"BWXfHXn3","testing_regress_ecl_analyzers_corporate_tmp_readme.md":"KZkrkEkE","testing_regress_ecl_readme.md":"DWXau1FL","tools_esdlcmd_readme.md":"D3nMaAu6","tools_esp-api_readme.md":"WcTRZBDR","tools_tagging_readme.md":"CdzS0CPr"}
diff --git a/index.html b/index.html
new file mode 100644
index 00000000000..56b280b0cb7
--- /dev/null
+++ b/index.html
@@ -0,0 +1,24 @@
+
+
+
+
+
+ HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
// Mount / to ./www directory
+auto ret = svr.set_mount_point("/", "./www");
+if (!ret) {
+ // The specified base directory doesn't exist...
+}
+
+// Mount /public to ./www directory
+ret = svr.set_mount_point("/public", "./www");
+
+// Mount /public to ./www1 and ./www2 directories
+ret = svr.set_mount_point("/public", "./www1"); // 1st order to search
+ret = svr.set_mount_point("/public", "./www2"); // 2nd order to search
+
+// Remove mount /
+ret = svr.remove_mount_point("/");
+
+// Remove mount /public
+ret = svr.remove_mount_point("/public");
cpp
// User defined file extension and MIME type mappings
+svr.set_file_extension_and_mimetype_mapping("cc", "text/x-c");
+svr.set_file_extension_and_mimetype_mapping("cpp", "text/x-c");
+svr.set_file_extension_and_mimetype_mapping("hh", "text/x-h");
The followings are built-in mappings:
Extension
MIME Type
txt
text/plain
html, htm
text/html
css
text/css
jpeg, jpg
image/jpg
png
image/png
gif
image/gif
svg
image/svg+xml
ico
image/x-icon
json
application/json
pdf
application/pdf
js
application/javascript
wasm
application/wasm
xml
application/xml
xhtml
application/xhtml+xml
NOTE: These the static file server methods are not thread safe.
// Send a final status without reading the message body.
+svr.set_expect_100_continue_handler([](const Request &req, Response &res) {
+ return res.status = 401;
+});
httplib::Client cli("httpbin.org");
+
+auto res = cli.Get("/range/32", {
+ httplib::make_range_header({{1, 10}}) // 'Range: bytes=1-10'
+});
+// res->status should be 206.
+// res->body should be "bcdefghijk".
Brotli compression is available with CPPHTTPLIB_BROTLI_SUPPORT. Necessary libraries should be linked. Please see https://github.com/google/brotli for more detail.
This is a high level description of the data obfuscation framework defined in system/masking/include. The framework includes a platform component, the engine, that exposes obfuscation logic supplied by dynamically loaded plugin libraries. The framework can be used to obfuscate sensitive data, such as passwords and PII. Possible use cases including preventing trace output from revealing sensitive values and partially masking values in situations where a user needs to see "enough" of a value to confirm its correctness without seeing all of the value (e.g., when a user is asked to confirm the last four digits of an account number).
File
Contents
datamasking.h
Declares the core framework interfaces. Most are used by the engine and plugins.
datamaskingengine.hpp
Defines the engine component from the core engine interface.
datamaskingplugin.hpp
Defines a set of classes derived from the core plugin interfaces that can be used to provide rudimentary obfuscation. It is expected that the some or all classes will be subclassed, if not replaced outright, in new plugins.
datamaskingshared.hpp
Defines utilities shared by engine and plugin implementations.
A domain is a representation of the obfuscation requirements applicable to a set of data.
Consider that the data used to represent individuals likely differs between countries. With different data, requirements for obfuscation may reasonably be expected to vary. Assuming that requirements do change between countries, each country's requirements could logically constitute a separate domain. The capacity to define multiple domains does not create a requirement to do so.
Requirements can change over time. To support this, a domain can be seen as a collection of requirement snapshots where each snapshot defines the complete set of requirements for the domain at a point in time. Snapshots are referenced by unique version numbers, which should be sequential starting at 1.
Obfuscation is always applied based on a single snapshot of a domain's requirements.
Domains are represented in the framework interface as text identifiers. Each distinct domain is identified by at least one unique identifier.
A masker is a provider of obfuscation for a single snapshot of a domain's requirements. There are three masking operations defined in this framework. Each instance decides which of the three it will support, and how it will support them. The three operations are:
maskValue obfuscates individual values based on snapshot-defined meanings. For example, a value identified as a password might require complete obfuscation anywhere it appears, or an account number may require complete obfuscation in some cases and partial obfuscation in others.
maskContent obfuscates a variable number of values based on context provided by surrounding text. For example, the value of an HTTP authentication header or the text between <Password> and </Password> might require obfuscation. This operation can apply to both structured and unstructured text content.
maskMarkupValue obfuscates individual values based on their locationa within an in-memory representation of a structured document, such as XML or JSON. For example, the value of element Password might require obfuscation unconditionally, while the value of element Value might require obfuscation only if a sibling element named Name exists with value password. This operation relies on the caller's ability to supply context parsed from structured content.
A masker may be either stateless or stateful. With a stateless masker, identical input will produce identical output for every requested operation. A stateful masker, however, enables its user to affect operation outputs (i.e., identical input may not produce identical output for each operation).
Maskers are represented in the framework interface using IDataMasker, with IDataMaskerInspector providing access to less frequently used information about the domain.
A profile is a stateless masker. Each instance defines the requirements of one or more snapshots of a single domain.
Snapshots are versioned. Each profile declares a minimum, maximum, and default version. Masker operations apply to the default version. Other declared versions may be accessed using a stateful context, which the profile can create on demand.
Each instance must support at least one version of a domain's requirements. Whether an instance supports more than one version depends on the implementation and on user preference. A domain can be viewed as a collection of one or more profiles where each profile defines a unique set of requirement snapshots applicable to the same underlying data.
Refer to IDataMaskingProfile (extending IDataMasker) and IDataMaskingProfileInspector (extending IDataMaskerInspector) for additional information.
A context is a stateful masker. Instantiated by and tightly coupled to a profile, it provides some user control over how masking operations are completed.
For a profile supporting multiple versions, the requirements of a non-default version may be applied.
Custom properties, defined by a profile, may be managed.
valuetype-set is a pre-defined property used to select the group of value types that may be masked by any operation.
rule-set is a pre-defined property used to select the group of rules that will be applied for maskContent requests.
Profile implementations may define additional properties as needed. For example, one might define mask-pattern to override the default obfuscation pattern to avoid replicating (and complicating) configurations just to change the appearance of obfuscated data.
Trace output produced by operation requests, including errors and warnings encountered during request processing, can be controlled per context. This does not affect the operation output, per-se, but provides compatibility with transactional trace output control.
Refer to IDataMaskingProfileContext (extending IDataMasker) and IDataMaskingProfileContextInspector (extending IDataMaskerInspector) for additional information.
Each snapshot defines one or more value types. A value type is a representation of the requirements pertaining to a particular concept of a domain datum. Requirements include:
instructions for identifying occurrences based on contextual clues found in a body of text; and
instructions for applying obfuscation to an identified value.
A Social Security Number, or SSN, is a U.S.-centric datum that requires obfuscation and for which a value type may be defined. Element names associated with an SSN may include, but are not limited to, SSN and SOCS; the value type is expected to identify all such names used within the domain. SSN occurrences are frequently partially masked, with common formats being to mask only the first four or the last five digits of the nine digit number; the value type defines which formats are available besides the default of masking all characters.
Value types are represented in the framework using IDataMaskingProfileValueType. They are accessed through the IDataMaskerInspector interface.
A mask style describes how obfuscation is applied to a value. It is always defined in the context of a value type, and a value type may define multiple.
A value type is not required to define any mask styles. If none are defined, all value characters are obfuscated. If the requested mask style is not defined, the default obfuscation occurs; the value type will not attempt to guess which of the defined styles is appropriate.
Mask styles are represented in the framework using IDataMaskingProfileMaskStyle. They are accessed through the IDataMaskingProfileValueType interface.
A rule contains the information necessary to locate at least one occurrence of a value type datum to be obfuscated. It is always defined in the context of a value type, and a value type may define multiple.
maskValue requests do not use rules. maskContent requests rely on rules for locating affected values, with the relationship between a profile and its rules an implementation detail. maskMarkupValue requests, when implemented, may also use rules or may take an entirely different approach.
Rules are represented in the framework as an abastract concept that cannot be inspected individually. Inspection may be used to establish the presence of rules, but not to examine individual instances.
The combination of a shared library and entry point function describes a plugin. The input to a plugin is a property tree describing one or more profiles. The output of an entry point function is an iterator of profiles. The profiles created by the function may all be associated with the same domain, but this is not required.
Plugin results are represented in the framework using IDataMaskingProfileIterator.
An engine is the platform's interface to obfuscation. It loads domains by loading one or more plugins. Plugins yield profiles, from which domains are inferred.
Once configured with at least one domain, a caller can obtain obfuscation in multiple ways:
As an instance of a stateless masker, an engine can provide obfuscation. The default version of the default profile of the default domain will be used, and no custom context properties can be used. This is the simplest integration, appropriate for use when the host process implicitly trusts that the default configuration is sufficient.
An engine may be used to obtain a stateless profile for a specific domain. If no version is requested, the default profile of the requested domain will be used. If a version is specified, the domainprofile supporting the requested version is used. This usage supports host processes that are aware of domains and need to use specific instances.
An engine may be used to obtain a context tied to a specific version of a domain. The default domain may be requested with an empty string. The default version may be requested by passing 0. With a context, a caller may manipulate further the environment for obfuscation requests.
Engines are represented in the framework using IDataMaskingEngine (extending IDataMasker) and IDataMaskingEngineInspector for additional information.
This section assumes a context is in use. Absent a context, only the default state of the default version of a profile can be used.
The framework reserves multiple custom property names, which are described in subsequent subsections. It also allows implementations to define additional properties using any non-reserved name.
Suppose an implementation defines a property to override the default obfuscation pattern. Let's call this property default-pattern. A caller could interrogate a masker to know if default-pattern is accepted. If accepted, setProperty("default-patterns", "#") can be used to register an override.
All custom properties, whether defined in the framework or in third party libraries, are managed using a generic context interface. This interface includes:
The abstraction includes the concept of sets of related value types. All value types should be assigned membership in at least one set. One set should be selected by default. The custom context property valuetype-set is used to select a different set.
The rationale for this is an expectation that certain data always requires obfuscation. A password, for example, would never not require obfuscation. Other data may only require obfuscacation in certain situations, such as when required by individual customer agreements. Callers should not be required to complete additional steps to act on data that must always be obfuscated, and should not need to know which data falls in which category.
The set name "*" is reserved to select all value types regardless of their defined set membership. This mechanism is intended to assist with compatibility checks, and should be used with care in other situations.
Use acceptsProperty to determine whether a snapshot recognizes a custom property name. Use usesProperty to learn if the snapshot includes references to the custom property name. Acceptance implies awareness of the set (and what it represents) even if selecting it will not change the outcome of any masking request. Usage implies that selecting the set will change the outcome of at least one masking operation.
The property valuetype-set is used to select a named set. A check of this property addresses whether the concept of a set is accepted or used by the snapshot.
For improved compatibility checks, implementations are encouraged to reserve additional property names that enable checks for individual set names. For each set name, foo, the included implementations report property valuetype-set:foo as used.
The included implementations allow membership in either a default, unnamed set or in any number of named sets. Absent a contextual request for a named set, the unnamed set is selected by default. With a contextual set request, the members of the named set are selected in addition to members of the unnamed set. Members of the unnamed set are never not selected.
Unnamed set membership cannot be defined explicitly. Because this set is always selected, assigning a value type to both a named and the unnamed set is redundant - the type will be selected whether the named set is requested or not.
Example 1a: default value type
profile:
+ valueType:
+ - name: type1
Value typetype1 belongs to the default, unnamed value type set. Property valuetype-set is accepted, but not used; an attempt to use it will change no results.
Extending the previous example, five types and four sets are in use. Property valuetype-set is both accepted and used, as it can now impact results. Properties valuetype-set:set1, valuetype-set:set2 and valuetype-set:set3 are also both accepted and used, but are intended only for use with compatibility checks.
This table shows which types are selected for each requested set:
Continuing the previous example, the profile has declared acceptance of two additional sets using properties valuetype-set:set4 and valuetype-set:set5. Selection of these sets will not change results, but a caller might accept acceptance of a set as sufficient to establish compatibility.
The abstraction includes the concept of sets of related rules. All rules should be assigned membership in at least one set. One set should be selected by default. The custom context property rule-set is used to select a different set.
The rationale for this is similar to yet different than that of value type sets. Instead of one set of rules that are always selected, backward compatibility with a proprietary legacy implementation requires that the default set be replaced by an alternate collection.
The set name "*" is reserved to select all rules regardless of their defined set membership. This does not override constraints imposed by the current value type set. This mechanism is intended to assist with compatibility checks, and should be used with care in other situations.
Use acceptsProperty to determine whether a snapshot recognizes a custom property name. Use usesProperty to learn if the snapshot includes references to the custom property name. Acceptance implies awareness of the set (and what it represents) even if selecting it will select no rules. Usage implies that selecting the set will select at least one rule.
The property rule-set is used to select a named set. A check of this property addresses whether the concept of a set is accepted or used by the snapshot.
For improved compatibility checks, implementations are encouraged to reserve additional property names that enable checks for individual set names. For each set name, foo, the included implementations report property rule-set:foo as used.
The included implementations allow membership in any number of named sets as well as a default, unnamed set. Absent a contextual request for a named set, the unnamed set is selected by default. With a contextual set request, only the members of the requested set are selected.
Unlike value type sets, the unnamed set is not always selected. Because of this difference, a rule may be assigned explicit membership in the unnamed set. This is optional for rules that belong only to the unnamed set and is required for rules meant to be selected as part of the unnamed set and one or more named sets.
The rules described here are intentionally incomplete, showing only what is necessary for this example. Four rules are defined:
The rule named foo implicitly belongs to the unnamed set.
The second rule explicitly belongs to the unnamed set.
The third rule explicitly belongs to the unnamed set and to set1.
The fourth rule belongs to set1.
property rule-set is both accepted and used. Properties rule-set: and rule-set:set1 are both accepted and used, intended for use with compatibility checks. Property rule-set:set2 is accepted.
Given a single domain datum, obfuscate the content "as needed". The framework anticipates two interpretations of "as needed", either conditional or unconditional:
Conditional means that values of a named type are only obfuscated if the named type is known by and selected in the snapshot. This usage assumes that the profile is the authoritative source for obfuscation requirements. The profile's consumer may ask for any value to be obfuscated, but the profile is not required to act.
Unconditional means that all value obfuscation requests will result in obfuscation. The named type is not required to be known by or selected in the snapshot. This usage assumes that the profile's consumer is the authoritative source for obfuscation requirements. If a consumer asks for a value to be obfuscated, the profile must act.
Each plugin will define its own interpretation of "as needed". The API distinction between conditional and unconditional is a hint intended to guide implementations capable of both, and should be ignored by implementations that are not.
The buffer, offset, and length parameters define what is assumed to be a single domain datum. That is, every character in the given character range is subject to obfuscation.
The valueType parameter determines if obfuscation is required, with conditionally controlling whether an unrecognized valueType is ignored (true) or forced to obfuscate (false).
The maskStyle parameter may be used to affect the nature of obfuscation to be applied. If the parameter names a defined mask style, that obfuscation format is applied. If the parameter does not name a define mask style, the value type's default obfuscation format is applied.
The canMaskValue method of IDataMasker can be used to establish whether a snapshot supports this operation. A result of true indicates that the plugin is capable of performing obfuscation in response to a request. Whether a combination of input parameters exists that results in obfuscation depends on the definition of the snapshot.
The hasValueType method of IDataMaskerInspector can be used to check if a given name is selected in the snapshot. The reserved name "*" may be used to detect unconditional obfuscation support when available in the snapshot. Names defined yet unselected in the snapshot are not directly detectable, but can be detected by comparing results of using multiple context configurations.
The name "*" is reserved by the abstraction for compatibility checks.
To confirm the availability of a specific mask style first requires confirmation of the value type to which the mask style is related. Use queryValueType to obtain the defined instance and, from the result, call queryMaskStyle to confirm the existence of the mask stye. For each of these, corresponding get... methods are defined to obtain new references to the objects. The methods are declared by IDataMaskerInspector.
The included implementations are inherently conditional. Obfuscation depends on matching valueType with a selected value type instance in the snapshot, where an instance is selected if it belongs to the currently selected value type set.
Unconditional obfuscation is enabled in profiles that include a value type instance named "*". If an instance with this name is selected by the currently selected value type set, all values for undefined value types will be obfuscated. Values for defined but unselected value types will not be obfuscated.
A value of type "foo" will be obfuscated when type "foo" is selected by the currently selected value type set.
A value of type "foo" will be obfuscated when type "foo" is undefined and type "*" is selected by the currently selected value type set.
A value of type "foo" will not be obfuscated when type "foo" is defined but unselected by the currently selected value type set.
This snippet declares two value types, with two total sets. The table shows which values will be conditionally obfuscated based on a the combination of valueType and the currently selected value type set.
valueType
Selected Set
Obfuscated
type1
N/A
Yes
type1
set1
Yes
type2
N/A
No
type2
set1
Yes
typeN
N/A
No
typeN
set1
No
"typeN" in the table represents any named type, excluding the reserved "*", not explicitly defined in the profile.
Extending the previous example with a third value type, values of unknown type are now obfuscated. Values of known but unselected type remain unobfuscated.
valueType
Selected Set
Obfuscated
type1
N/A
Yes
type1
set1
Yes
type2
N/A
No
type2
set1
Yes
typeN
N/A
Yes
typeN
set1
Yes
The name reserved by the abstraction to detect support for unconditional obfuscation is the same name reserved by the implementation to define that support.
"typeN" in the table represents any named type, excluding the reserved "*", not explicitly defined in the profile.
The buffer, offset, and length parameters define what is assumed to be a blob of content that may contain zero or more occurrences of domain data. The blob is expected to contain sufficient context allowing a snapshot's rules to locate any included occurrences.
The context parameter optionally influences which rules are selected in a snapshot, while the contentType parameter offers a hint as to the blob's data format. Each snapshot rule may be assigned a format to which it applies. Operation requests may be optimized by limiting the number of selected rules applied by describing the format. Selected rules not assigned a format will be applied in all requests.
There is no equivalent to the unconditional mode offered by maskValue. Content obfuscation is always conditional on matching rules.
The canMaskContent method of IDataMasker can be used to establish whether a snapshot supports this operation. A result of true indicates that the plugin is capable of performing obfuscation in response to a request. Whether a combination of input parameters exists that results in obfuscation depends on the definition of the snapshot.
The hasRule method of IDataMaskerInspector indicates if at least one rule is selected in the snapshot for a given content type. Rules not associated with any content type may be interrogated using an empty string.
It is not possible to discern any information about the selected rules, such as their associated value types or the cues used to apply them in a buffer.
The groundwork exists for two types of rules, serial and parallel. Serial rules, as the name implies, are evaluated sequentially. Parallel rules, on the other hand, are evaluated concurrently. Concurrent evaluation is expected to be more efficient than sequential, but no concurrent implementation is provided.
The included sequential implementation should be viewed as a starting point. Each rule defines a start token and an optional end token. For each occurrence of the start token in the blob, a corresponding search for an end token (newline if omitted), is performed. When both parts are found, the content between is obfuscated using the associated value type's default mask style.
Traversing a potentially large blob of text once per rule is inefficient. A concurrent implementation is in development to improve performance and capabilities. A domain originated using the original implementation and migrated to the new implementation illustrates one domain implemented by multiple plugins and, by extension, multiple profiles.
Where maskValue obfuscates individual values based on what the caller believes the value to be, and maskContent obfuscates any number of values it identifies based on cues found in surrounding text, maskMarkupValue obfuscastes individual values based on cues not provided to it.
Given a value and a relative location within a structure document, an implementation must decide whether obfuscation is required, may be required, or is not required. If required, it can obfuscate immediately. If not required, and can return immediately. If it might be required, the implementation must request the context it needs from the caller in order to make a final determination.
A request to mask the content of an element named Value might depend on the content of a sibling element named Name. If value of Value probably does not require obfuscation when the value of Name is city, but almost certainly does require obfuscation when the value of Name is password. When processing the request for Value, an implementation must ask for the value of a sibline named Name to decide whether or not to obfuscate the data.
There is no equivalent to the unconditional mode offered by maskValue. Markup value obfuscation presumes that the caller is traversing a structured document without awareness of which values require obfuscation. If the caller knows which values require obfuscation, it should use maskValue on those specific values.
The canMaskMarkupValue method of IDataMasker can be used to establish whether a snapshot supports this operation. A result of true indicates that the plugin is capable of performing obfuscation in response to a request. Whether a combination of input parameters exists that results in obfuscation depends on the definition of the snapshot.
The getMarkupValueInfo method of IDataMaskerInspector determines if the value of an element or attribute requires obfuscation. If required, the info parameter will describe on completion how to perform the obfuscation using maskValue instead of maskMarkupValue. This is done to optimize performance by eliminating the need to reevaluate rules to re-identify a match.
Assme a value type representing passwords is defined. If given an element value containing a URL with an embedded password, obfuscation is required. But most likely not for the entire value. In addition to identifying the value type associated with value, the offset and length of the embedded substring requiring obfuscation are supplied; use of maskValue with these three values and no mask style will apply the default obfuscation only to the embedded password. Alternatively, if a mask style is supplied, use of maskValue with this style and the original value offset and length should obfuscate only the embedded password.
A callback interface is required for all calls to getMarkupValueInfo. The interface is only used when additional document context is required to make a determination about the requested value. In these cases, the checks are proximity-dependent and cannot be performed as part of load-time compatibility checks, when no document is available.
Use of runtime compatibility checks may be unavoidable in some circumstances, but reliance on this in every script is inefficient. It may also devalue trace output by omitting messaging required to debug an issue, a problem that might only be found when said messaging is needed.
The requirements parameter represents a standardized set of runtime compatibility checks to be applied. By default, each check must pass to establish compatibility. Explicitly optional checks may be included to detect conditions the caller is prepared to work around.
This can test which values will or won't be affected by maskValue, and which obfuscation styles are available. It can test which rule sets will impact maskContent. It specifically excludes maskMarkupValue checks that depend on the proximity of one value to another.
Returning to the earlier SSN example, a caller might require that an SSN value type will be obfuscated. It might also want to use a particilar mask style, but may be prepared for its absence. A caller may prefer to mask the first four digits but may accept masking the last five instead, or may omit certain trace messages.
compatibility:
+ - context:
+ - domain: optional text
+ version: optional number
+ property:
+ - name: required text
+ value: required text
+ accepts:
+ - name: required text
+ presence: one of "r", "o", or "p"
+ uses:
+ - name: required text
+ presence: one of "r", "o", or "p"
+ operation:
+ - name: one of "maskValue", "maskContent", or "maskMarkupValue"
+ presence: one of "r", "o", or "p"
+ valueType
+ - name: required text
+ presence: one of "r", "o", or "p"
+ maskStyle:
+ - name: required text
+ presence: one of "r", "o", or "p"
+ Set:
+ - name: required text
+ presence: one of "r", "o", or "p"
+ rule:
+ - contentType: required text
+ presence: one of "r", "o", or "p"
The requirements parameter of checkCompatibility must be either a compatibility element or the parent of a collection of compatibility elements. Each instance may contain at most one context element, three operation elements (one per operation), and a variable number of accepts, uses, valueType, or rule elements.
The context element describes the target of an evaluation. It may include a domain identifier, or assume the default domain. It may include a version number, or assume the default snapshot of the domain. It may specify any number of custom context properties to select various profile elements in the snapshot.
The accepts and uses elements identify custom context property names either accepted or used within a snapshot. Acceptance indicates some part of a snapshot has declared an understanding of the named property. Usage indicates the some part of a snapshot will react to the named value being set. Usage implies acceptance.
To improve compatibility check capabilities, a snapshot may synthesize properties that, if set, will have no effect. Specifically, a caller may be more interested to know if a particular set name (for either value type or rule set membership) is used than to know that an unidentified set name is used.
The Operations element identifies which operations must be supported. Its purpose is to enforce availability of an operation when no more explicit requirements are given (for example, one may require maskContent without also requiring the presence of rules for any given content type). When more explicit requirements are given, the attributes of this element are redundant. If all attributes are redundant, the element may be omitted.
The valueType element describes all optional and mandatory requirements for using maskValue with a single value type name and optional mask style name. Set membership requirements can also be evaluated, which is most useful when compatibility/context/property/@name is "*".
The rule element describes all optional and mandatory requirements for using maskContent with a single content type value. Unlike evaluable value type set membership, rule set membership requirements cannot be evaluated.
In all elements described as accepting @presence, the acceptable values are
r: the check is required to pass
o: the check is optional
p: the check is prohibited from passing
+
+
+
+
\ No newline at end of file
diff --git a/system/masking/plugins/datamasker/readme.html b/system/masking/plugins/datamasker/readme.html
new file mode 100644
index 00000000000..512ecf77416
--- /dev/null
+++ b/system/masking/plugins/datamasker/readme.html
@@ -0,0 +1,47 @@
+
+
+
+
+
+ HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
This implements a plugin library producing instances of IDataMaskingProfileIterator to be used by an IDataMaskingEngine instance. The library is written in C++, and this section is organized according to the namespace and class names used.
Standard implementation of IDataMaskingProfileContext and IDataMaskingProfileContextInspector tightly coupled to a specific version of a specific profile instance. It manages custom properties that are accepted by the associated profile, and rejects attempts to set those not accepted. The rejection is one way of letting the caller know that something it might deem essential is not handled by the profile.
Depends on CProfile, an abstract implementation of IDataMaskingProfile and IDataMaskingProfileInspector.
Implementation of IDataMaskingProfileMaskStyle providing customized masked output of values. Configuration options include:
@maximumVersion is an optional version number, needed only when the style does not apply to the maximum version of the value type in which it is defined.
@minimumVersion is an optional version number, needed only when the style does not apply to the minumum version of the value type in which it is defined.
@name is a required unique identifier for the style.
@overrideDefault is an optional Boolean flag controlling whether the style should replace the default style for the value type in which it is defined.
@pattern is an optional character sequence (one or more characters in length) that will be used to mask value content.
Am extension of CMaskStyle, this is intended to partially mask values, such as account numbers or telephone numbers, without assuming knowledge of the values. Configuration is generally expressed as:
@action what to do, either keep or mask (default); and
@location where to do it, either first or last (default); and
@count how many characters to examine, a positive integer value; and
@characters type of characters to be considered, either numbers, letters, alphanumeric, or all (default)
All four values may be omitted when only the inherited options are needed. If any of the four values are given, count is required and the other three are optional.
A value substring containing at most @count instances of the character class denoted by @characters is identified. Character classes are ASCII numeric characters (numbers), ASCII alphabetic characters (letters), ASCII alphabetic and numeric characters (alphanumeric), or any characters (all).
The value substring is at the start of the value when @location is first, and at the end of the value when @location is last.
The value substring is masked when @action is mask. All of the value with the exception of the substring is masked when @action is keep.
A base class for rule implementations, this defines standard rule properties without providing any content knowledge for use with maskContent.
Configuration includes:
@contentType is an optional, user-defined, label describing the content format to which the rule should be applied. maskContent requests that indicate a content type should apply all rules that match the type in addition to all rules that omit a type.
memberOf is an optional and repeating element identifying user-defined set labels. Only rules with membership in the requested set are considered; in the absence of an explictly requested set, a default set with an empty name is implied.
memberOf/@name is a required user-defined set label.
rule instances will frequently be specific to a content format. For example, a rule applied to XML markup value may include dependencies on XML syntax which are not applicable when masking JSON content. Rules specific to a markup format can be associated with that format using @contentType. Requests to mask content of a type will apply all rules without a @contentType value or with a matching value, and will exclude all rules with a non-matching value.
An extension of CRule, this identifies content substrings to be masked based on matching start and end tokens in the content buffer. For each occurrence of a configured start token that is balanced by a corresponding configured end token, the characters between the tokens are masked.
@endToken is an optional character sequence expected to immediately follow a content substring to be masked. This might be an XML element end tag, or a terminating double quote for a JSON value. A newline, i.e., \n, is assumed if omitted or empty, for masking values such as HTTP headers.
@matchCase is an optional Boolean flag controlling whether token matches are case sensitive (true) or insensitivie (false). Matches are case insensitive by default. The implementation is based on ASCII data; case sensitive comparisons of similarly encoded non-ASCI text can work, but case insensitive comparisons are not supported.
@startToken is a required character sequence expected to precede a content substring to be masked.
No content type knowledge is implied by this class. An instance with @contentType of xml does not inherently know how to find values in XML markup. The defined tokens must include characters such as < and > to match an element name and quotes to match attribute values.
An implementation of IDataMaskingProfileIterator used for transforming a configuration property tree into a collection of profiles to be returned by a library entry point function. Created profiles are of the same class, identified by the template parameter profile_t.
Configuration includes:
profile An optional and repeating element defining a profile. The content of this element depends on profile_t.
If and only if profile is not included as a child of the configuration node, the node itself will be treated as a configuration for profile_t. The name of the configuration node is not required to be profile.
A concrete extension of CProfile, this manages value types and rules, providing a default implementation of maskValue but leaving other operations to subclasses.
Template parameters are:
valuetype_t identifies the concrete implementation of IDataMaskingProfileValueType to be instantiated during configuration.
rule_t identifies the concrete implementation of rules to be managed. Creation is assumed to be the responsibility of the value type.
context_t identifies the concrete implementation of IDataMaskingProfileContext to be instantiated on demand.
Configuration includes:
@defaultVersion is an optional version number identifying the version to be used when version 0 is requested. Omission or 0 implies the maximum version (which is configured before this value).
@domain is the required default identifier used to select this profile.
legacyDomain is an optional and repeating element identifying alternate identifiers by which the profile may be selected. This enables domain identifier naming conventions to be changed without breaking pre-existing references.
legacyDomain/@id is a required identifier of this profile.
@maximumVersion is an optional version number identifying the highest version supported by the configuration. Omission or 0 implies a value matching the minimum version (whether implicitly or explicitly defined).
@minimumVersion is an optional version number identifying the lowest version supported by the configuration. Omission or 0 implies 1.
@name is an optional label for the instance. If given, the value is used in trace output.
property is an optional and repeating element describing a custom context property recognized by the profile. The profile can declare awareness of properties without explicit use of them.
property/@name is a required context property name.
property/@minimumVersion is an optional version number indicating the lowest profile version aware of the property. Omission or 0 implies the profile minimum.
property/@maximumVersion is an optional version number indicating the highest profile version aware of the property. Omisssion or 0 implies the profile maximum.
valueType is an optional and repeating element describing the value types defined by the profile. Element content depends on the value of valuetype_t.
For profiles supporting a single version, value type names must be unique. For profiles supporting multiple versions, value type names may be repeated but must be unique for each version. To illustrate, consider this snippet:
In the example, foo and bar are each defined twice. The redefinition of foo is acceptable because each instance applies to a different version. The redefinition of bar is invalid because both instances claim to apply to version 2.
The value type name * is reserved by this class. A profile that includes a value type named * supports unconditional value masking. maskValue requests specifying unknown value type names can fall back to this special type and force masking for these values. Without this type, maskValue masks what it thinks should be masked; with this type, it trusts the caller to request masking for only those values known to require it. Value types included in unselected value type sets are known to the profile and, as such, can be used to prevent specific typed values from ever being masked.
The requirement of a type definition instead of a simpler flag is to enable the definition of mask styles. It has the side effect of enabling the definition of rules. In theory, an entire profile could be defined using a single value type. This may make sense in some cases, but not in all. For example, a partial mask style intended for use with U.S. Social Security numbers could be inappropriately applied to a password. Use care when configuring this type.
An extension of TProfile that adds support for maskContent. It is assumed by maskContent that each applicable rule must be applied serially, i.e., one after the other, using an bool applyRule(buffer, length, context) interface.
Template parameters are unchanged from TProfile.
Configuration options are unchanged from TProfile.
An implementation of IDataMaskingProfileValueType that manages mask styles and creates rules for the profile.
Template parameters are:
maskstyle_t identifies the concrete implementation of IDataMaskingProfileMaskStyle to be instantiated during configuration.
rule_t identifies the concrete implementation of rules to be created during configuration.
Configuration includes:
@maximumVersion is an optional version number, needed only when the type does not apply to the maximum version of the profile in which it is defined.
@minimumVersion is an optional version number, needed only when the type does not apply to the minumum version of the profile in which it is defined.
maskStyle is an optional and repeating element describing mask styles defined by the type. Element content depends on the value of maskstyle_t.
memberOf is an optional and repeating element identifying user-defined set labels. All value types that are not explicitly assigned set membership are considered for all masking requests. Value types assigned set membership are considered only when one of their assigned sets is selected with the request context.
memberOf/@name is a required user-defined set label.
@name is a required unique identifier for the type.
rule is an optional and repeating element describing rules defined by the type. Element content depends on the value of rule_t.
For value types supporting a single version, mask style names must be unique. For types supporting multiple versions, style names may be repeated but must be unique for each version. To illustrate, consider this snippet:
In the example, foo and bar are each defined twice. The redefinition of foo is acceptable because each instance applies to a different version. The redefinition of bar is invalid because both instances claim to apply to version 2.
This section describes the entry point functions exported by the shared library. The library must export one function, and may export multiple functions.
The description of each entry point will identify which of the previously described classes is used to represent the returned collection of profiles. If a templated class is identified, the template parameters are also listed. Refer to the class descriptions for additional information.
The purpose of this plugin is to provide authentication and authorization capabilities for HPCC Systems users, with the credentials passed via valid JWT tokens.
The intention is to adhere as closely as possibly to the OpenID Connect (OIC) specification, which is a simple identity layer on top of the OAuth 2.0 protocol, while maintaining compatibility with the way HPCC Systems performs authentication and authorization today. More information about the OpenID Connect specification can be found at https://openid.net/specs/openid-connect-core-1_0.html.
One of the big advantages of OAuth 2.0 and OIC is that the service (in this case, HPCC Systems) never interacts with the user directly. Instead, authentication is performed by a trusted third party and the (successful) results are passed to the service in the form of a verifiable encoded token.
Unfortunately, HPCC Systems does not support the concept of third-party verification. It assumes that users -- really, any client application that operates as a user, including things like IDEs -- will submit username/password credentials for authentication. Until that is changed, HPCC Systems won't be able to fully adhere to the OIC specification.
We can, however, implement most of the specification. That is what this plugin does.
NOTE: This plugin is not available in a Windows build.
The plugin is called by the HPCC Systems esp process when a user needs to be authenticated. That call will contain the user's username and either a reference to a session token or a password. The session token is present only for already-authenticated users.
If the session token is not present, the plugin will call a JWT login service (also known as a JWT login endpoint) with the username and password, plus a nonce value for additional security.
That service authenticates the username/password credentials. If everything is good, the service constructs an OIC-compatible token that includes authorization information for that user and returns it to the plugin. The token is validated according to the OIC specification, including signature verification.
Note that token signature verification requires an additional piece of information. Tokens can be signed with a hash-based algorithm or with a public key-based algorithm (the actual algorithm used is determined by the JWT service). To verify either kind of algorithm, the plugin will need either the secret hash key or the public key that matches what the JWT service used. That key is read by the plugin from a file, and the file is determined by a configuration setting (see below). It is possible to change the contents of that file without restarting the esp process. Note, though, that the plugin may not notice that the file's contents have changed for several seconds (changes do not immediately take effect).
HPCC Systems uses a well-defined authorization scheme, originally designed around an LDAP implementation. That scheme is represented within the token as JWT claims. This plugin will unpack those claims and map to the authorization checks already in place within the HPCC Systems platform.
OIC includes the concept of refresh tokens. Refresh tokens enable a service to re-authorize an existing token without user intervention. Re-authorization typically happens due to a token expiring. Tokens should have a relatively short lifetime -- e.g. 15-30 minutes -- to promote good security and also give administrators the ability to modify a user's authorization while the user is logged in. This plugin fully supports refresh tokens by validating token lifetime at every authorization check and calling a JWT refresh service (also known as a JWT refresh endpoint) when needed. This largely follows the OIC specification.
Initial authentication: As stated above, the esp process will gather the username/password credentials instead of a third party, then send those credentials off to another service. In a true OIC configuration, the client process (the esp process) never sees user credentials and relies on an external service to gather them from the user.
The request made to the JWT login service is a POST HTTP or HTTPS call (depending on your configuration) containing four items in JSON format; example:
The most obvious outcome of this implementation is that a custom service/endpoint needs to be available. Or rather two services: One to handle the initial user login and one to handle token refreshes. Neither service precisely handles requests and replies in an OIC-compatible way, but the tokens themselves are OIC-compatible, which is good. That allows you to use third-party JWT libraries to construct and validate those tokens.
Several items must be defined in the platform's configuration. Within configmgr, the jwtsecmgr Security Manager plugin must be added as a component and then modified according to your environment:
The URL or unique name of this HPCC Systems cluster, used as the client_id in token requests
Full URL to the JWT Login Endpoint (should be HTTPS, but not required)
Full URL to the JWT Refresh Endpoint (should be HTTPS, but not required)
Boolean indicating whether to accept self-signed certificates for those endpoints; defaults to false
Secrets vault key/name or subdirectory under /opt/HPCCSystems/secrets/esp in which the JWT key used for the chosen signature algorithm is stored; defaults to "jwt-security"
Default permission access level (either "Full" or "None"); defaults to "Full"
Default workunit scope access level (either "Full" or "None"); defaults to "Full"
Default file scope access level (either "Full" or "None"); defaults to "Full"
Only the first three items have no default values and must be supplied.
Once the jwtseccmgr component is added, you have to tell other parts of the system to use the plugin. For user authentication and permissions affecting features and workunit scopes, you need to add the plugin to the esp component. Instructions for doing so can be found in the HPCC Systems Administrator's Guide manual (though the manual uses the htpasswd plugin as an example, the process is the same).
If you intend to implement file scope permissions then you will also need provide Dali information about the JWT plugin. In configmgr, within the Dali Server component, select the LDAP tab. Change the authMethod entry to secmgrPlugin and enter "jwtsecmgr" as the authPluginType. Make sure checkScopeScans is set to true.
This plugin supports all authorizations documented in the HPCC Systems® Administrator's Guide with the exception of "View Permissions". Loosely speaking, the permissions are divided into three groups: Feature, Workunit Scope, and File Scope.
Feature permissions are supported exactly as documented. A specific permission would exist as a JWT claim, by name, with the associated value being the name of the permission. For example, to grant read-only access to ECL Watch, use this claim:
{ "SmcAccess": "Read" }
File and workunit scope permissions are handled the same way, but different from feature permissions. The claim is one of the Claim constants in the tables below, and the associated value is a matching pattern. A pattern can be simple string or it can use wildcards (specifically, Linux's file globbing wildcards). Wildcards are not typically needed.
Cleanup Parameter of Regression Suite run and query sub-command
The cleanup parameter has been introduced to allow the user to automatically delete the workunits created by executing the Regression Suite on their local system.
It is an optional argument of the run and query sub-command.
A custom logging system also creates log files for each execution of the run and query sub-command that contains information about the workunit deletion.
The Parquet plugin test suite is a subset of tests in the HPCC Systems regression suite. To run the tests:
Change directory to HPCC Platform/testing/regress.
To run the entire Parquet test suite:
Note: Some Parquet tests require initialization of Parquet files. Use the ./ecl-test setupcommand to initialize these files before running the test suite.
These commands can be run on any cluster, including hthor or Roxie like the example below.
This project focuses on the development of a comprehensive test suite for the recently integrated Parquet plugin within the HPCC Systems platform. The objective is to thoroughly evaluate the plugin's functionality, performance, and robustness across different scenarios and configurations. The key deliverables include defining and implementing various test cases, fixing any identified bugs, and providing extensive documentation.
The test suite will evaluate all data types supported by ECL and Arrow, as well as file operations, various compression formats, and schema handling. Additionally, the test suite will measure the plugin's performance across different HPCC components and hardware configurations, conduct stress tests to identify potential bottlenecks and bugs, and compare Parquet to other file formats used in the ecosystem, such as JSON, XML, and CSV.
Tests all available Arrow compression types: Snappy, GZip, Brotli, LZ4, ZSTD, Uncompressed Compares performance and file sizes for different compression options
The test suite generally uses key files located in HPCC-Platform/testing/regress/ecl/key, with a ".xml" extension, to evaluate test outcomes. These files store the expected results for comparison. However, some Parquet tests do not rely on key files, and alternative evaluation methods are used in those cases.
+
+
+
+
\ No newline at end of file
diff --git a/testing/regress/ecl/analyzers/corporate/tmp/README.html b/testing/regress/ecl/analyzers/corporate/tmp/README.html
new file mode 100644
index 00000000000..6eac39e6eb9
--- /dev/null
+++ b/testing/regress/ecl/analyzers/corporate/tmp/README.html
@@ -0,0 +1,24 @@
+
+
+
+
+
+ HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
The esdl utility tool aids with creating and managing ESDL-based and Dynamic ESDL services on an HPCC cluster. It consists of several different commands.
To generate output from an ESDL definition:
xml Generate XML from ESDL definition.
+ecl Generate ECL from ESDL definition.
+xsd Generate XSD from ESDL definition.
+wsdl Generate WSDL from ESDL definition.
+java Generate Java code from ESDL definition.
+cpp Generate C++ code from ESDL definition.
+monitor Generate ECL code for result monitoring / differencing
+monitor-template Generate a template for use with 'monitor' command
+
To manage ESDL and DESDL services:
publish Publish ESDL Definition for ESP use.
+list-definitions List all ESDL definitions.
+get-definition Get ESDL definition.
+delete Delete ESDL Definition.
+bind-service Configure ESDL based service on target ESP (with existing ESP Binding).
+list-bindings List all ESDL bindings.
+unbind-service Remove ESDL based service binding on target ESP.
+bind-method Configure method associated with existing ESDL binding.
+unbind-method Remove method associated with existing ESDL binding.
+get-binding Get ESDL binding.
+manifest Build a service binding or bundle from a manifest file.
+bind-log-transform Configure log transform associated with existing ESDL binding.
+unbind-log-transform Remove log transform associated with existing ESDL binding.
+
The sections below cover the commands in more detail.
The manifest command creates an XML configuration file for an ESDL ESP from an input XML manifest file. The type of configuration output depends on the manifest file input and on command-line options.
A manifest file is an XML-formatted template combining elements in and outside of the manifest's urn:hpcc:esdl:manifest namespace. Recognized elements of this namespace control the tool while all other markup is copied to the output. The goal of using a manifest file with the tool is to make configuring and deploying services easier:
The manifest file format abstracts some of the complexity of the actual configuration.
By allowing you to include external files like ESDL Scripts and XSLTs into the ouput, you can store and maintain them separately in your repo.
The result of running the manifest tool on a manifest file is an XML artifact suitable for use with the ESDL ESP. Supported output includes:
binding: The output is an ESDL binding that may be published to dali.
bundle: The output is an ESDL bundle file that may be used to launch an ESP in application mode.
The tool is permissive and flexible, copying through most markup to the output. Recognized elements in the manifest namespace may be treated differently. They are only required in order to take advantage of the automated processing and simplified format of the manifest file. This example highlights the recommended usage of manifest elements to use the tool's capabilities. Although you could replace some of the elements below with verbatim bundle or binding output elements we won't cover that usage here.
<em:Manifest> is the required root element. By default the tool outputs a bundle, though you may explicitly override that on the command line or by providing an @outputType='binding' attribute.
<em:ServiceBinding> is valid for both bundle and binding output. It is necessary to enable recognition of <em:Scripts>, and <em:Transform> elements.
<em:EsdlDefinition> is relevant only for bundle output. It is necessary to enable element order preservation and recognition of <em:Include> as a descendant element.
<em:Include> causes external file contents to be inserted in place of the element. The processing of included files is context dependent; the parent of the <em:Include> element dictates how the file is handled. File inclusion facilitates code reuse in a configuration as code development environment.
<em:Scripts> and <em:Transform> trigger preservation of element order for all descendent elements and enable <em:Include> recognition.
XML element ordering is significant to proper ESDL script, XSLT, and ESXDL content processing. The platform's IPropertyTree implementation, used when loading an artifact, does not preserve order. Output configuration files must embed order-sensitive content as text as opposed to XML markup. The tool allows configuration authors to create and maintain files as XML markup, which is easy to read. It then automates the conversion of the XML markup into the embedded text required by an ESP.
These elements may create artifact content, change the tool's behavior, or both. When used as intended, none of these elements will appear in the generated output:
Required root element of all manifest files that create output:
bundle output is created by default. It can be made explicit by setting either/or:
command line option --output-type to bundle
manifest property @outputType to bundle.
binding output is created when either/or:
command line option --output-type is binding
manifest property @outputType is binding.
Attribute
Required?
Value
Description
@outputType
N
string
A hint informing the tool which type of output to generate. The command line option --output-type may supersede this value to produce a different output. A bundle manifest is a superset of a binding manifest and may logically be used to create either output type. A binding manifest, as a subset of a bundle, cannot be used to create a valid bundle.
@xmlns[:prefix]
Y
string
The manifest namespace urn:hpcc:esdl:manifest must be declared. The default namespace prefix should not be used unless all other markup is fully qualified.
Recommended child of Manifest that creates ESDL binding content and applies ESDL binding-specific logic to descendent content:
A Binding output element is always created containing attributes of Manifest as required. See the sections below for details of the attributes referenced by the tool and how they're output.
A child of Binding named Definition is created with attributes @id and @esdlservice.
Recognition of <em:Scripts> elements is enabled.
Recognition of <em:Transform> elements is enabled.
While possible to omit the <em:ServiceBinding> element and instead embed a complete Binding tree in the manifest, it is discouraged because you lose the benefit of the processing described above.
There are three categories of attributes that can be defined on the <em:ServiceBinding> element:
Standard attributes needed for setup
Service-specific or binding-specific attributes
Auxillary attributes recommended for read-only reference
First are the standard attributes used to setup and define the binding:
Attribute
Required?
Value
Usage
@esdlservice
Yes
string
- Name of the ESDL service to which the binding is bound. Output on the Binding/Definition element. - Also used to generate a value for Definition/@id for bundle type output.
Note that Definition/@id is not output by the tool for binding output as it would need to match the ESDLDefinitionId passed on command line esdl bind-service call, which may differ between environments.
Second are binding-specific or service-specific attributes. This is an open-ended category where most future attributes will belong. Any attribute not in the other two categories is included here and output on the Binding/Definition element:
Attribute
Required?
Value
Usage
@auth_feature
No
string
Used declare authorization settings if they aren't present in the ESDL Definition, or override them if they are. Additional documentation on this attribute is forthcoming.
@returnSchemaLocationOnOK
No
Boolean
When true, a successful SOAP response (non SOAP-Fault) will include the schema location property. False by default.
@namespace
No
string
String specifying the namespace for all methods in the binding. May contain variables that are replaced with their values by the ESP during runtime: - ${service} : lowercase service name - ${esdl-service} : service name, possibly mixed case - ${method} : lowercase method name - ${esdl-method} : method name, possibly mixed case - ${optionals} : comma-delimited list of all optional URL parameters included in the method request, enclosed in parentheses - ${version} : client version number
Finally are auxillary attributes. These should be thought of as read-only or for reference, and it is not recommended that you set these in the manifest. They are set by the system when publishing to dali using esdl bind-service, and they are present on binding configurations retrieved from the dali. If you set these attributes in the manifest, the values will be overwritten when running esdl bind-service. Alternately, if you run as an esdl application, these values aren't set by default and don't have a material effect on the binding, but they may appear in the trace log.
Attribute
Required?
Value
Usage
@created
No
string
Timestamp of binding creation
@espbinding
No
string
Set to match the id. Otherwise the value is unset and unused.
@espprocess
No
string
Name of the ESP process this binding is running on
@id
No
string
Runtime name of the binding. When publishing to dali the value is [ESP Process].[port].[ESDL Service]. When not present in the manifest a default value is generated of the form [@esdlservice]_desdl_binding
@port
No
string
Port on the ESP Process listening for connections to the binding.
Optional element that imports the contents of another file into the output in place of itself. The outcome of the import depends on the context in which this element is used. See EsdlDefinition, Scripts, and Transform for more information.
Attribute
Required?
Value
Usage
@file
Yes
file path
Full or partial path to an external file to be imported. If a partial path is outside of the tool's working directory, the tool's command line must specify the appropriate root directory using either -I or --include-path.
Any XSLTs or ESDL Scripts written inline in a manifest file will have XML escaping applied where required to generate valid XML. If an XSLT contains any text content or markup that needs to be preserved as-is (no XML escaping applied) then be sure to use an <em:Include> operation. Included files are inserted into the output as-is, with the exception of encoding nested CDATA markup. If the included file will be inside a CDATA section on output, then any CDATA end markup in the file will be encoded as ]]]]><![CDATA[> to prevent nested CDATA sections or a prematurely ending a CDATA section.
Optional repeatable element appearing within an <em:ServiceBinding> element that processes child elements and creates output expected for an ESDL binding:
The <em:Scripts> element is replaced on output with <Scripts>. Then all content is enclosed in a CDATA section after wrapping it with a new Scripts element. That new <Scripts> element contains namespaces declared by the input <em:Scripts> element. The input <em:Scripts foo="..." xmlns:bar="..."><!-- content --></em:Scripts> becomes <Scripts><![CDATA[<Scripts xmlns:bar="..."><!-- content --></Scripts>]]></Scripts>.
The manifest <em:Include> element is recognized to import scripts from external files. The entire file, minus leading and trailing whitespace, is imported. Refrain from including files that contain an XML declaration.
Optional repeatable element appearing within an <em:ServiceBinding> element that processes child elements and creates output expected for an ESDL binding:
All content is enclosed in a CDATA section. The input <em:Transform> <!-- content --> </em:Transform> becomes <Transform><![CDATA[<!-- content -->]]></Transform>.
The manifest <em:Include> element is recognized to import transforms from external files. The entire file, minus leading and trailing whitespace, is imported. Refrain from including files that contain an XML declaration.
Usage:
+
+esdl manifest <manifest-file> [options]
+
+Options:
+ -I | --include-path <path>
+ Search path for external files included in the manifest.
+ Use once for each path.
+ --outfile <filename>
+ Path and name of the output file
+ --output-type <type>
+ When specified this option overrides the value supplied
+ in the manifest attribute Manifest/@outputType.
+ Allowed values are 'binding' or 'bundle'.
+ When not specified in either location the default is
+ 'bundle'
+ --help Display usage information for the given command
+ -v,--verbose Output additional tracing information
+ -tcat,--trace-category <flags>
+ Control which debug messages are output; a case-insensitive
+ comma-delimited combination of:
+ dev: all output for the developer audience
+ admin: all output for the operator audience
+ user: all output for the user audience
+ err: all error output
+ warn: all warning output
+ prog: all progress output
+ info: all info output
+ Errors and warnings are enabled by default if not verbose,
+ and all are enabled when verbose. Use an empty <flags> value
+ to disable all.
+
The esdl manifest command reads the manifest, processes statements in the urn:hpcc:esdl:manifest namespace and generates an output XML file formatted to the requirements of the ESDL ESP. This includes wrapping included content in CDATA sections to ensure element order is maintained and replacing urn:hpcc:esdl:manifest elements as required.
An example output of each type -bundle and binding- is shown below. The examples use the sample manifest above as input plus these included files:
ESDL Directory Location: The tool gathers the directory of the ESDL files from an environment configuration variable; gets the install path from jutil library and appends /componentfiles/esdl_files/.
Server and Port Defaults: When using the test command, if the server and port are not specified, the tool defaults to interacting with an ESP instance located at http://127.0.0.1:8010.
Custom ESDL Directory Argument: An additional argument could be introduced to allow users to specify the directory of the ESDL files directly in the command.
Template Request Generation: A feature could be added to generate template XML or JSON requests. This would simplify the process of filling out requests by providing a pre-structured template.
Credential Prompts: The tool could be expanded to prompt for the username and password upon a 401 Unauthorized response.
Selective Response Extraction: Another potential feature is to allow extraction of specific tags from the response using XPath expressions. This would make it easier to parse and analyze responses.
+
+
+
+
\ No newline at end of file
diff --git a/tools/tagging/README.html b/tools/tagging/README.html
new file mode 100644
index 00000000000..ad199593a5e
--- /dev/null
+++ b/tools/tagging/README.html
@@ -0,0 +1,36 @@
+
+
+
+
+
+ Tagging new versions | HPCC Platform
+
+
+
+
+
+
+
+
+
+
+
+
+
+
The following process should be followed when tagging a new set of versions.
Upmerge all changes between candidate branches for the different versions
You can set the all environment variable to a subset of the projects (e.g. export all=hpcc) if there are no changes in the other repositories. The only effect for projects that are upmerged with no changes will be that they gain an empty merge transaction. If multiple people are merging PRs to different repositories it may be safer to upmerge all projects.
If you have merged changes onto a point-release branch you would normally create a new rc before going gold. If the change was trivial (e.g. removing an unwanted file) then you can use the --ignore option to skip that step.
Creating a new rc for an existing point release:
This normally happens after cherry-picking a late fix for a particular version, which has already been merged into the .x candidate branch.
