Skip to content

Commit

Permalink
Merge pull request #98 from picodata/97-special-placeholders
Browse files Browse the repository at this point in the history 
97 special placeholders
  • Loading branch information
@DmitryTravyan
DmitryTravyan authored Mar 24, 2023
2 parents 9761bbb + 12a2747 commit 632c54d
Show file tree
Hide file tree
Showing 10 changed files with 212 additions and 40 deletions.
2 changes: 1 addition & 1 deletion Cargo.lock
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "genin"
version = "0.4.13"
version = "0.4.14"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
Expand Down
24 changes: 12 additions & 12 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ Download and unzip the archive for the desired architecture.

Universal executable:
```shell
curl -sLO https://binary.picodata.io/repository/raw/genin/bin/genin-0.4.13-x86_64-musl.tar.gz
tar -xvf genin-0.4.13-x86_64-musl.tar.gz ; sudo install genin /usr/local/bin/
curl -sLO https://binary.picodata.io/repository/raw/genin/bin/genin-0.4.14-x86_64-musl.tar.gz
tar -xvf genin-0.4.14-x86_64-musl.tar.gz ; sudo install genin /usr/local/bin/
```

---
Expand Down Expand Up @@ -84,11 +84,11 @@ sudo yum install -y genin
2. If you want to install `rpm` packages directly without
adding our repository.
```shell
sudo rpm -i https://binary.picodata.io/repository/yum/el/8/x86_64/os/genin-0.4.13-1.el8.x86_64.rpm
sudo rpm -i https://binary.picodata.io/repository/yum/el/8/x86_64/os/genin-0.4.14-1.el8.x86_64.rpm
```
RHEL 7.x, CentOS 7.x
```shell
sudo rpm -i https://binary.picodata.io/repository/yum/el/7/x86_64/os/genin-0.4.13-1.el7.x86_64.rpm
sudo rpm -i https://binary.picodata.io/repository/yum/el/7/x86_64/os/genin-0.4.14-1.el7.x86_64.rpm
```

---
Expand Down Expand Up @@ -116,7 +116,7 @@ sudo apt install -y genin
2. Downloading and installing the package directly:
```shell
curl -sLO https://binary.picodata.io/repository/raw/genin/deb/genin-0.4.13.amd64.deb && sudo dpkg -i genin-0.4.13.amd64.deb
curl -sLO https://binary.picodata.io/repository/raw/genin/deb/genin-0.4.14.amd64.deb && sudo dpkg -i genin-0.4.14.amd64.deb
```
---
Expand All @@ -142,7 +142,7 @@ sudo apt install -y genin
2. Downloading and installing the package directly:
```shell
curl -sLO https://binary.picodata.io/repository/raw/genin/deb/genin-0.4.13.amd64.deb && sudo dpkg -i genin-0.4.13.amd64.deb
curl -sLO https://binary.picodata.io/repository/raw/genin/deb/genin-0.4.14.amd64.deb && sudo dpkg -i genin-0.4.14.amd64.deb
```
---
Expand All @@ -162,8 +162,8 @@ brew install genin
Use the following command to grab and install Genin in macOS (10.10+) wihtout
homebrew:
```shell
curl -L https://binary.picodata.io/repository/raw/genin/apple/genin-0.4.13-darwin-amd64.zip -o genin-0.4.13-darwin-amd64.zip
unzip genin-0.4.13-darwin-amd64.zip -d ~/bin/
curl -L https://binary.picodata.io/repository/raw/genin/apple/genin-0.4.14-darwin-amd64.zip -o genin-0.4.14-darwin-amd64.zip
unzip genin-0.4.14-darwin-amd64.zip -d ~/bin/
```
> **Note:** The application can then be found under the `~/bin` directory.
> Make sure the directory is in your `$PATH`.
Expand All @@ -181,8 +181,8 @@ brew install genin@0.3.8
#### Windows
Use the following command to grab and install Genin in Windows 7 64 bit or newer:
```shell
curl.exe -L https://binary.picodata.io/repository/raw/genin/windows/genin-0.4.13-darwin-amd64.zip -o genin-0.4.13-windows-amd64.zip
unzip.exe genin-0.4.13-windows-amd64.zip -d %HOME%/.cargo/bin/
curl.exe -L https://binary.picodata.io/repository/raw/genin/windows/genin-0.4.14-darwin-amd64.zip -o genin-0.4.14-windows-amd64.zip
unzip.exe genin-0.4.14-windows-amd64.zip -d %HOME%/.cargo/bin/
```
> **Note:** The application can then be found under the `.cargo/bin` folder inside
> your user profile folder. Make sure it is in your `%PATH%`.
Expand Down Expand Up @@ -261,8 +261,8 @@ hosts:
# in this example, both hosts are in the same cloud data center
config: # (optional) begin binary and http port, by default 8081, 3031
# ports can be defined on all levels
http: 8081 # (optional) http port to start counting from
binary: 3031 # (optional) binary port to start counting from
http_port: 8081 # (optional) http port to start counting from
binary_port: 3031 # (optional) binary port to start counting from
hosts:
- name: host-1 # (mandatory) hostname or domain name
config:
Expand Down
22 changes: 11 additions & 11 deletions README.ru.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ Genin уже заранее скомпилирован под разные ар

Универсальный исполняемый файл:
```shell
curl -sLO https://binary.picodata.io/repository/raw/genin/bin/genin-0.4.13-x86_64-musl.tar.gz
tar -xvf genin-0.4.13-x86_64-musl.tar.gz ; sudo install genin /usr/local/bin/
curl -sLO https://binary.picodata.io/repository/raw/genin/bin/genin-0.4.14-x86_64-musl.tar.gz
tar -xvf genin-0.4.14-x86_64-musl.tar.gz ; sudo install genin /usr/local/bin/
```

---
Expand Down Expand Up @@ -87,11 +87,11 @@ sudo yum install -y genin
2. Так же вы можете установить пакет `rpm` напрямую без добавления нашего репозитория.
RHEL 8.x, CentOS 8.x, Rockylinux 8.x, recent Fedora version
```shell
sudo rpm -i https://binary.picodata.io/repository/yum/el/8/x86_64/os/genin-0.4.13-1.el8.x86_64.rpm
sudo rpm -i https://binary.picodata.io/repository/yum/el/8/x86_64/os/genin-0.4.14-1.el8.x86_64.rpm
```
RHEL 7.x, CentOS 7.x
```shell
sudo rpm -i https://binary.picodata.io/repository/yum/el/7/x86_64/os/genin-0.4.13-1.el7.x86_64.rpm
sudo rpm -i https://binary.picodata.io/repository/yum/el/7/x86_64/os/genin-0.4.14-1.el7.x86_64.rpm
```
> **Note:** будьте внимательны, так как при выборе не правильной версии ос могут быть ошибки
> при установке `rpm`
Expand All @@ -118,7 +118,7 @@ sudo apt install -y genin

2. Загрузкой и установкой пакета напрямую:
```shell
curl -sLO https://binary.picodata.io/repository/raw/genin/deb/genin-0.4.13.amd64.deb && sudo dpkg -i genin-0.4.13.amd64.deb
curl -sLO https://binary.picodata.io/repository/raw/genin/deb/genin-0.4.14.amd64.deb && sudo dpkg -i genin-0.4.14.amd64.deb
```

---
Expand All @@ -143,7 +143,7 @@ sudo apt install -y genin

2. Загрузкой и установкой пакета напрямую:
```shell
curl -sLO https://binary.picodata.io/repository/raw/genin/deb/genin-0.4.13.amd64.deb && sudo dpkg -i genin-0.4.13.amd64.deb
curl -sLO https://binary.picodata.io/repository/raw/genin/deb/genin-0.4.14.amd64.deb && sudo dpkg -i genin-0.4.14.amd64.deb
```

---
Expand All @@ -163,8 +163,8 @@ brew install genin
Для установки без помощи homebrew используйте следующие команды для загрузки и установки
Genin на macOS (10.10+):
```shell
curl -sLO https://binary.picodata.io/repository/raw/genin/osx/genin-0.4.13-x86_64-macosx.tar.gz
unzip genin-0.4.13-darwin-amd64.zip -d ~/bin/
curl -sLO https://binary.picodata.io/repository/raw/genin/osx/genin-0.4.14-x86_64-macosx.tar.gz
unzip genin-0.4.14-darwin-amd64.zip -d ~/bin/
```

---
Expand All @@ -188,8 +188,8 @@ brew install genin@0.3.8
Используйте следующие команды для скачивания и установки Genin на операционных системах
Windows 7 64 и новее.
```shell
curl.exe -sLO https://binary.picodata.io/repository/raw/genin/win/genin-0.4.13-win64.zip
unzip.exe genin-0.4.13-win64.zip -d %HOME%/.cargo/bin/
curl.exe -sLO https://binary.picodata.io/repository/raw/genin/win/genin-0.4.14-win64.zip
unzip.exe genin-0.4.14-win64.zip -d %HOME%/.cargo/bin/
```
> **Note:** Genin будет распакован в директорию `.cargo/bin` которая находится в домашнем
> каталоге важего пользователя. Перед использованием приложения пожалуйста удостоверьтесь
Expand All @@ -199,7 +199,7 @@ unzip.exe genin-0.4.13-win64.zip -d %HOME%/.cargo/bin/
```
genin --version
```
Если вы видите сообщение `genin 0.4.13` значит установка прошла успешно.
Если вы видите сообщение `genin 0.4.14` значит установка прошла успешно.

---
## Руководство по использованию
Expand Down
107 changes: 103 additions & 4 deletions src/task.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,14 @@ pub mod serde_genin;
pub mod vars;

use log::info;
use regex::RegexBuilder;
use serde_yaml::{Mapping, Value};
use std::collections::HashMap;
use std::convert::TryFrom;
use std::error::Error;
use std::fmt;
use std::fs::File;
use std::io::Read;
use std::io::{Read, Write};

use crate::error::{GeninError, GeninErrorKind};
use crate::task::cluster::fs::{TryMap, IO, UPGRADE_YAML};
Expand Down Expand Up @@ -50,6 +52,74 @@ pub fn run_v2() -> Result<(), Box<dyn Error>> {
std::env::var("RUST_LOG").unwrap_or_else(|_| "warn".into())
);

// Cluster init comments

let comments = [
(
"replicasets_count".to_string(),
"# How many masters we want, by default equal 1".to_string(),
),
(
"replication_factor".to_string(),
"# Number of replicas in replicaset, default for router 0".to_string(),
),
(
"address".to_string(),
"# Host or instance address (may be IP or URI)".to_string(),
),
(
"http_port".to_string(),
"# Specify http port to start counting from".to_string(),
),
(
"binary_port".to_string(),
"# Specify binary port to start counting from".to_string(),
),
(
"weight".to_string(),
"# Vshard replicaset weight (matters only if `vshard-storage` role is enabled)"
.to_string(),
),
(
"zone".to_string(),
"# Zone parameter for ansible cartridge playbook".to_string(),
),
(
"config".to_string(),
"# Config with arbitrary key-values pairs".to_string(),
),
(
"mode".to_string(),
"# Failover mode (stateful, eventual, disabled)".to_string(),
),
(
"state_provider".to_string(),
"# What is serve failover (stateboard, stateful)".to_string(),
),
(
"stateboard_params".to_string(),
"# Params for chosen in state_provider failover type".to_string(),
),
(
"ansible_user".to_string(),
"# Username under which the ansible will connect to the servers".to_string(),
),
(
"ansible_password".to_string(),
"# Ansible user password".to_string(),
),
(
"cartridge_cluster_cookie".to_string(),
"# Cookie for connecting to the administrative console of the instances".to_string(),
),
(
"cartridge_package_path".to_string(),
"# Path to the application package".to_string(),
),
]
.into_iter()
.collect::<HashMap<String, String>>();

// The idea of the first step of creating a task:
// - create FsInteration
// - map FsInteration as:
Expand All @@ -71,7 +141,37 @@ pub fn run_v2() -> Result<(), Box<dyn Error>> {
})
})?
.print_input()
.serialize_input()?;
.try_map(|io| {
if let IO {
input: Some(cluster),
output: Some(mut file),
} = io
{
let mut text = serde_yaml::to_string(&cluster)
.map_err(|err| GeninError::new(GeninErrorKind::Deserialization, err))?;

for (k, v) in comments {
let comment =
RegexBuilder::new(&format!(r"(?P<key>^.+{}:[ ]*[^#<> ]+)$", k))
.multi_line(true)
.build()
.unwrap();
text = comment
.replace_all(&text, &format!("$key {}", v))
.to_string();

file.write(text.as_bytes()).map_err(|err| {
GeninError::new(GeninErrorKind::Deserialization, err)
})?;
}

return Ok(IO {
input: Some(()),
output: Some(()),
});
}
Err(GeninError::new(GeninErrorKind::EmptyField, "TODO"))
})?;
}
Some(("build", args)) => {
IO::from(args)
Expand All @@ -81,7 +181,6 @@ pub fn run_v2() -> Result<(), Box<dyn Error>> {
args.get_flag("force"),
)?
.deserialize_input::<Cluster>()?
//.map_err(|err|ClusterError::from(err))?
.print_input()
.try_map(|IO { input, output }| {
Inventory::try_from(&input).map(|inventory| IO {
Expand Down Expand Up @@ -168,7 +267,7 @@ pub fn run_v2() -> Result<(), Box<dyn Error>> {

pub trait Validate {
type Type: fmt::Debug + Default + 'static;
type Error: fmt::Debug;
type Error: fmt::Debug + ToString;

fn validate(bytes: &[u8]) -> Result<Self::Type, Self::Error>;

Expand Down
44 changes: 42 additions & 2 deletions src/task/cluster.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ pub mod topology;
use clap::ArgMatches;
use indexmap::IndexMap;
use log::{debug, trace};
use regex::{Captures, RegexBuilder};
use serde::ser::SerializeStruct;
use serde::{Deserialize, Serialize};
use serde_yaml::Value;
Expand Down Expand Up @@ -411,14 +412,53 @@ impl Validate for Cluster {
type Error = serde_yaml::Error;

fn validate(bytes: &[u8]) -> Result<Self::Type, Self::Error> {
serde_yaml::from_slice(bytes)
serde_yaml::from_str(&check_placeholders(bytes)?)
}

fn whole_block(bytes: &[u8]) -> String {
String::from_utf8(bytes.to_vec()).unwrap()
}
}

pub fn check_placeholders(slice: &[u8]) -> Result<String, serde_yaml::Error> {
let text = String::from_utf8_lossy(slice).to_string();
let reg = RegexBuilder::new(r"(?P<key>^.+:) +(<<.*>>) *(?:# *([^#:]+)$)*")
.multi_line(true)
.build()
.map_err(|err| serde::de::Error::custom(err))?;
let captures = reg.captures_iter(&text).collect::<Vec<Captures>>();

if captures.is_empty() {
return Ok(text);
}

let mut result = format!("\n{}", text);

for c in captures {
let placeholder = c.get(2).map(|r| r.as_str().to_string()).unwrap_or_default();
let comment = c
.get(3)
.map(|r| r.as_str().to_string())
.unwrap_or("Please replace or remove!".to_string());

result = reg
.replace(
&result,
&format!(
"$key Err({})",
format!(
"The placeholder {} was not replaced! {}",
placeholder, comment
)
.as_error()
),
)
.to_string();
}

Err(serde::de::Error::custom(&result))
}

impl Cluster {
pub fn spread(self) -> Self {
Self {
Expand Down Expand Up @@ -715,7 +755,7 @@ impl std::fmt::Debug for InvalidCluster {
Value::Null => {}
Value::Mapping(_) => {}
_ => {
formatter.write_str("\nvars:: ")?;
formatter.write_str("\nvars: ")?;
formatter.write_str("Vars must be a mapping".as_error().as_str())?;
formatter.write_str("\n")?;
}
Expand Down
Loading

0 comments on commit 632c54d

Please sign in to comment.