first commit

README.md

@@ -1,2 +1,20 @@
 # ai-cluster
-Start clusters in virtualbox VMs
+This repo contains notes that I used to setup 3-nodes clusters in centos 7 running in virtualbox VMs, all versions are the up-to-date stable ones at the time of writting.
+
+The list included are:
+1. [Hadoop V3.2.1](docs/HADOOP.md)
+
+2. [Zookeeper V3.5.7](docs/ZOOKEEPER.md)
+
+3. [Spark V2.4.5](docs/SPARK.md) 
+
+4. [HBase V2.2.3](docs/HBASE.md)
+
+5. [KAFKA V2.12-2.4.0](docs/KAFKA.md)
+
+6. [ETCD V3.4.4](docs/ETCD.md)
+
+7. [Cassandra V3.11.6](docs/CASSANDRA.md)
+
+
+For kubernetes cluster, please refer to another repo: [ai-install-k8s](https://github.com/aiden-dai/ai-install-k8s.git)

REDIS.md

@@ -0,0 +1,192 @@
+# Install Redis Cluster
+
+## References
+
+https://redis.io/topics/cluster-tutorial
+
+
+## Machines
+
+machine | ip address | os | hostname | Role
+-|-|-|-|-
+master | 192.168.56.150 | centos 7 | aiden-master | Master
+slave1 | 192.168.56.151 | centos 7 | aiden-slave1 | Worker
+slave2 | 192.168.56.152 | centos 7 | aiden-slave2 | Worker
+
+For your first tests it is strongly suggested to start a six nodes cluster with three masters and three slaves.
+
+## Pre-requisites
+
+- ...
+
+```
+yum install tcl
+```
+
+## Installation
+
+```
+tar xzvf redis-5.0.7.tar.gz
+cd redis-5.0.7
+make
+make test
+make install
+```
+
+```
+[root@localhost redis-5.0.7]# make test
+  ...
+  99 seconds - integration/replication-psync
+  132 seconds - integration/replication
+
+\o/ All tests passed without errors!
+
+Cleanup: may take some time... OK
+make[1]: Leaving directory `/root/redis-5.0.7/src'
+```
+
+
+
+
+## Configuration
+
+- redis.conf
+
+
+
+## Start
+
+Running Redis
+-------------
+
+To run Redis with the default configuration just type:
+
+```
+redis-server
+redis-server /path/to/redis.conf
+```
+
+Start
+```
+                _._                                                  
+           _.-``__ ''-._                                             
+      _.-``    `.  `_.  ''-._           Redis 5.0.7 (00000000/0) 64 bit
+  .-`` .-```.  ```\/    _.,_ ''-._                                   
+ (    '      ,       .-`  | `,    )     Running in standalone mode
+ |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
+ |    `-._   `._    /     _.-'    |     PID: 18565
+  `-._    `-._  `-./  _.-'    _.-'                                   
+ |`-._`-._    `-.__.-'    _.-'_.-'|                                  
+ |    `-._`-._        _.-'_.-'    |           http://redis.io        
+  `-._    `-._`-.__.-'_.-'    _.-'                                   
+ |`-._`-._    `-.__.-'    _.-'_.-'|                                  
+ |    `-._`-._        _.-'_.-'    |                                  
+  `-._    `-._`-.__.-'_.-'    _.-'                                   
+      `-._    `-.__.-'    _.-'                                       
+          `-._        _.-'                                           
+              `-.__.-'                  
+```
+
+
+## Create cluster
+
+
+```
+mkdir redis-cluster
+cd redis-cluster
+mkdir 7000 7001 7002 7003 7004 7005
+```
+
+create a redis.conf in each folder
+```
+port 7000
+cluster-enabled yes
+cluster-config-file nodes.conf
+cluster-node-timeout 5000
+appendonly yes
+```
+
+start each servers
+```
+cd 7000
+redis-server redis.conf
+```
+
+create cluster
+```
+redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 \
+127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 \
+--cluster-replicas 1
+
+[root@localhost redis-cluster]# redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 \
+> 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 \
+> --cluster-replicas 1
+>>> Performing hash slots allocation on 6 nodes...
+Master[0] -> Slots 0 - 5460
+Master[1] -> Slots 5461 - 10922
+Master[2] -> Slots 10923 - 16383
+Adding replica 127.0.0.1:7004 to 127.0.0.1:7000
+Adding replica 127.0.0.1:7005 to 127.0.0.1:7001
+Adding replica 127.0.0.1:7003 to 127.0.0.1:7002
+...
+[OK] All 16384 slots covered.
+```
+
+
+```
+redis-cli -p 7000 cluster nodes
+```
+
+## Create cluster - Quickstart
+
+```bash
+cd utils/create-cluster
+
+# init
+./create-cluster start
+
+# Create cluster
+./create-cluster create
+
+# Stop cluster
+./create-cluster stop
+
+# clean
+./create-cluster clean
+```
+
+
+```
+[root@localhost create-cluster]# ./create-cluster create
+>>> Performing hash slots allocation on 6 nodes...
+Master[0] -> Slots 0 - 5460
+Master[1] -> Slots 5461 - 10922
+Master[2] -> Slots 10923 - 16383
+Adding replica 127.0.0.1:30005 to 127.0.0.1:30001
+Adding replica 127.0.0.1:30006 to 127.0.0.1:30002
+Adding replica 127.0.0.1:30004 to 127.0.0.1:30003
+...
+[OK] All 16384 slots covered.
+```
+
+## Check
+
+- check nodes
+```
+redis-cli -p 7000 cluster nodes
+```
+
+
+## Client
+
+
+```
+[root@localhost src]# ./redis-cli 
+127.0.0.1:6379> ping
+PONG
+127.0.0.1:6379> set foo bar
+OK
+127.0.0.1:6379> get foo
+"bar"
+
+```

docs/CASSANDRA.md

@@ -0,0 +1,104 @@
+# Install Cassandra Cluster
+
+## References
+
+http://cassandra.apache.org/doc/latest/getting_started/installing.html#prerequisites
+
+
+## Machines
+
+machine | ip address | os | hostname
+-|-|-|-
+node1 | 192.168.56.150 | centos 7 | aiden-master
+node2 | 192.168.56.151 | centos 7 | aiden-slave1
+node3 | 192.168.56.152 | centos 7 | aiden-slave2
+
+## Pre-requisites
+
+- JDK installed
+
+## Installation
+
+Download the tar file.
+```
+tar zxvf apache-cassandra-3.11.6-bin.tar.gz -C /usr/local
+mv /usr/local/apache-cassandra-3.11.6 /usr/local/cassandra-3.11.6
+```
+
+Add below to profile
+```
+export CASSANDRA_HOME=/usr/local/cassandra-3.11.6
+export PATH=$PATH:$CASSANDRA_HOME/bin
+```
+
+
+## Configuration
+On each of the nodes
+- Create directions
+```bash
+mkdir -p /root/cassandra/data
+mkdir -p /root/cassandra/commitlog
+mkdir -p /root/cassandra/saved_caches
+mkdir -p /root/cassandra/hints
+```
+
+- conf/cassandra.yaml
+
+```yaml
+cluster_name: 'Test Cluster'
+...
+seed_provider:
+
+    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
+      parameters:
+
+          - seeds: "192.168.56.150,192.168.56.151,192.168.56.152"
+...
+# For other node, change this accordingly
+listen_address: 192.168.56.150
+
+
+data_file_directories:
+    - /root/cassandra/data
+
+commitlog_directory: /root/cassandra/commitlog
+saved_caches_directory: /root/cassandra/saved_caches
+hints_directory: /root/cassandra/hints
+```
+## Start
+
+Run as Root
+```
+# foreground
+cassandra -f -R
+
+# background
+cassandra -R
+```
+
+## Check
+
+- nodetool status
+```
+[root@aiden-master conf]# nodetool status
+Datacenter: datacenter1
+=======================
+Status=Up/Down
+|/ State=Normal/Leaving/Joining/Moving
+--  Address         Load       Tokens       Owns (effective)  Host ID                               Rack
+UN  192.168.56.152  114.87 KiB  256          66.5%             e901e4e5-7b05-4ef6-9777-7bb5cb22701f  rack1
+UN  192.168.56.150  80.64 KiB  256          64.8%             2d390507-c8d7-4d24-8729-56c6527038aa  rack1
+UN  192.168.56.151  137.61 KiB  256          68.7%             e6cd3418-4024-4234-93d6-2eb3eec0ab65  rack1
+```
+
+
+## Client
+
+- cqlsh
+```
+[root@aiden-master bin]# cqlsh
+Connected to Test Cluster at 127.0.0.1:9042.
+[cqlsh 5.0.1 | Cassandra 3.11.6 | CQL spec 3.4.4 | Native protocol v4]
+Use HELP for help.
+cqlsh> 
+```