The IBMStreams/streamsx.dps GitHub repository is home to the Streams toolkit named DPS (Distributed Process Store). This Streams toolkit allows a simple way for the SPL, C++ and Java operators belonging to a single or multiple applications to share the application specific state information via an external K/V store. It does this via a collection of APIs that can be called from any part of the SPL, C++ and Java operator code.
Following are the external NoSQL K/V stores that can be configured to work with the DPS toolkit for the purpose of sharing application state in a distributed manner.
- Memcached
- Redis [version 2.x that doesn't have a built-in cluster feature]
- Cassandra
- IBM Cloudant
- HBase
- Mongo
- Couchbase
- Redis-Cluster [New cluster feature is available in Redis version 3 and above]
- Valkey
In March/2024, the Redis company switched the licensing for the Redis core code repository from the free of cost BSD license to for-fee proprietary licenses. This is in effect for any Redis version higher than 7.2.5. This prompted a large portion of the user and developer community, led by the Linux Foundation, to fork the Redis code v7.2.5 under the new name Valkey, retaining the BSD license. Valkey looks and works exactly like redis except for its name and its free of cost availability as open source.
DPS toolkit supports redis as before for the free versions up to v7.2.5 and any redis proprietary paid versions higher than v7.2.5. DPS toolkit creator Senthil Nathan completed the testing to ensure that DPS toolkit also supports Valkey v8.0.2 and higher. DPS toolkit users can install Valkey single server or a cluster with TLS or non-TLS and password or no-password options and use the DPS toolkit configuration as before with redis or redis-cluster or redis-cluster-plus-plus as the backend database name. No code changes will be required in the DPS enabled applications for them to work with Valkey.
This toolkit evolved from the early research work done at the IBM.T.J.Watson Research Center, Yorktown Heights, New York. The links below highlight the origins of this asset.
For an API description see the GitHub pages for this toolkit. The documentation is also available locally in the doc folder, after building the toolkit. For configuration hints and other information, see the Wiki pages
Other things to consider when working with this toolkit:
To build the toolkit perform the following steps:
- Clone the repository
git clone https://github.com/IBMStreams/streamsx.dps.git - Build the toolkit
make clean all
Prerequisuites
you need to have the following rpm packages installed on the build machine:
curl
curl-devel
lua
lua-devel
openldap-devel
openssl-devel
cyrus-sasl
cyrus-sasl-devel