Merge pull request #522 from WeBankFinTech/dev-1.0.1

[release] Prepare for release DSS-1.0.1
WeBankFinTech · Feb 24, 2022 · efda868 · efda868
2 parents 4c62c74 + 34229c4
commit efda868
Show file tree

Hide file tree

Showing 22 changed files with 179 additions and 162 deletions.
diff --git a/README-ZH.md b/README-ZH.md
@@ -81,23 +81,24 @@
 
  &nbsp; &nbsp; &nbsp; &nbsp;**如果有需要，也可以轻松集成新的数据应用系统，以替换或丰富 DSS 的数据应用开发流程。** [点我了解如何快速集成新的应用系统](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E5%BC%80%E5%8F%91%E6%96%87%E6%A1%A3/%E7%AC%AC%E4%B8%89%E6%96%B9%E7%B3%BB%E7%BB%9F%E6%8E%A5%E5%85%A5DSS%E5%BC%80%E5%8F%91%E6%8C%87%E5%8D%97.md)
 
-| 应用工具     | 描述                                                          | DSS0.X 版本要求   | DSS1.0 版本要求    | 版本状态 |
-| --------------- | -------------------------------------------------------------------- | --------- | ---------- | ------ |
-| [**DataApiService**](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E4%BD%BF%E7%94%A8%E6%96%87%E6%A1%A3/DataApiService%E4%BD%BF%E7%94%A8%E6%96%87%E6%A1%A3.md)  | 数据API服务。可快速将SQL脚本发布为一个 Restful 接口，对外提供 Rest 访问能力。    | 不支持 | >=1.0.0 | 已发布 |
-| [**Scriptis**](https://github.com/WeBankFinTech/Scriptis)   | 支持在线写 SQL、Pyspark、HiveQL 等脚本，提交给[Linkis](https://github.com/WeBankFinTech/Linkis)执行的数据分析 Web 工具。 | >=0.5.0 | >=1.0.0 | 已发布      |
-| [**Schedulis**](https://github.com/WeBankFinTech/Schedulis) | 基于 Azkaban 二次开发的工作流任务调度系统,具备高性能，高可用和多租户资源隔离等金融级特性。 | >=0.5.0 | >=1.0.0 | 已发布      |
-| **EventCheck**  | 提供跨业务、跨工程和跨工作流的信号通信能力。 | >=0.5.0 | >=1.0.0 | 已发布      |
-| **SendEmail**   | 提供数据发送能力，所有其他工作流节点的结果集，都可以通过邮件进行发送 | >=0.5.0 | >=1.0.0 | 已发布  |
-| [**Qualitis**](https://github.com/WeBankFinTech/Qualitis)   | 数据质量校验工具，提供数据完整性、正确性等数据校验能力 | >=0.5.0 | 1.0.1(**当前筹备版本**) |  **预计年前发版**      |
-| [**Streamis**](https://github.com/WeBankFinTech/Streamis)  | 流式应用开发管理工具。支持发布 Flink Jar 和 Flink SQL ，提供流式应用的开发调试和生产管理能力，如：启停、状态监控、checkpoint 等。 | 不支持 | 1.0.1(**当前筹备版本**) | **预计年前发版** |
-| [**Exchangis**](https://github.com/WeBankFinTech/Exchangis) | 支持对结构化及无结构化的异构数据源之间的数据传输的数据交换平台，即将发布的 Exchangis1.0，将与 DSS 工作流打通 | 不支持 | 规划在1.0.2 | **开发实现中**      |
-| [**Visualis**](https://github.com/WeBankFinTech/Visualis)   | 基于宜信开源项目 Davinci 二次开发的数据可视化 BI 工具，为用户在数据安全方面提供金融级数据可视化能力。 | >=0.5.0 | 规划在1.0.2 | **开发实现中**      |
-| [**Prophecis**](https://github.com/WeBankFinTech/Prophecis)     | 一站式机器学习平台，集成多种开源机器学习框架。Prophecis 的 MLFlow 通过 AppConn 可以接入到 DSS 工作流中。      | 不支持 | 规划在1.0.2 | **开发实现中** |
-| **UserManager** | 自动初始化一个 DSS 新用户所必须的所有用户环境，包含：创建 Linux 用户、各种用户路径、目录授权等。                 |  >=0.9.1 | 规划在1.0.2 | **开发实现中** |
-| **DolphinScheduler** | Apache DolphinScheduler，分布式易扩展的可视化工作流任务调度平台，支持一键将DSS工作流发布到 DolphinScheduler。    | 不支持 | 规划在1.1.0 | **开发实现中** |
-| **UserGuide**     | 包含帮助文档、新手指引、Dark模式换肤等。      | 不支持 | 规划在1.1.0 | **开发实现中** |
-| **DataModelCenter**     | 主要提供数仓规划、数据模型开发和数据资产管理的能力。数仓规划包含主题域、数仓分层、修饰词等；数据模型开发包含指标、维度、度量、向导式建表等；数据资产打通 **Apache Atlas**，提供**数据血缘**能力。      | 不支持 | 规划在1.2.0 | **开发实现中** |
-| **Airflow**     | 支持将 DSS 工作流发布到 Airflow 进行定时调度。      | >=0.9.1，尚未合并 | 不支持 | **尚无规划** |
+| 应用工具     | 描述                                                          | DSS0.X 兼容版本(推荐 DSS0.9.1)   | DSS1.0 兼容版本(推荐 DSS1.0.1)    |
+| --------------- | -------------------------------------------------------------------- | --------- | ---------- |
+| [**Linkis**](https://github.com/apache/incubator-linkis) | 计算中间件 Apache Linkis，通过提供 REST/WebSocket/JDBC/SDK 等标准接口，上层应用可以方便地连接访问 MySQL/Spark/Hive/Presto/Flink 等底层引擎. | >= Linkis0.5.0（**已发布**） | >= Linkis1.0.3（**已发布**） |
+| [**DataApiService**](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/zh_CN/%E4%BD%BF%E7%94%A8%E6%96%87%E6%A1%A3/DataApiService%E4%BD%BF%E7%94%A8%E6%96%87%E6%A1%A3.md)  | （DSS已内置的第三方应用工具）数据API服务。可快速将SQL脚本发布为一个 Restful 接口，对外提供 Rest 访问能力。    | 不支持 | >= DSS1.0.0（**已发布**）|
+| [**Scriptis**](https://github.com/WeBankFinTech/DataSphereStudio)   | （DSS 已内置的第三方应用工具）支持在线写 SQL、Pyspark、HiveQL 等脚本，提交给[Linkis](https://github.com/WeBankFinTech/Linkis)执行的数据分析 Web 工具。 | >= DSS0.5.0（**已发布**） | >= DSS1.0.0（**已发布**） |
+| [**Schedulis**](https://github.com/WeBankFinTech/Schedulis) | 基于 Azkaban 二次开发的工作流任务调度系统,具备高性能，高可用和多租户资源隔离等金融级特性。 | <= Schedulis0.6.1（**已发布**） | >= Schedulis0.6.2（**开发中**） |
+| **EventCheck**  | （DSS 已内置的第三方应用工具）提供跨业务、跨工程和跨工作流的信号通信能力。 | >= DSS0.5.0（**已发布**） | >= DSS1.0.0（**已发布**） |
+| **SendEmail**   | （DSS 已内置的第三方应用工具）提供数据发送能力，所有其他工作流节点的结果集，都可以通过邮件进行发送 | >= DSS0.5.0（**已发布**） | >= DSS1.0.0（**已发布**） |
+| [**Qualitis**](https://github.com/WeBankFinTech/Qualitis)   | 数据质量校验工具，提供数据完整性、正确性等数据校验能力 | >= Qualitis0.8.0（**已发布**） | >= Qualitis0.9.0（**开发中**） |
+| [**Streamis**](https://github.com/WeBankFinTech/Streamis)  | 流式应用开发管理工具。支持发布 Flink Jar 和 Flink SQL ，提供流式应用的开发调试和生产管理能力，如：启停、状态监控、checkpoint 等。 | 不支持 | >= Streamis0.1.0（**开发中**） |
+| [**Exchangis**](https://github.com/WeBankFinTech/Exchangis) | 支持对结构化及无结构化的异构数据源之间的数据传输的数据交换平台，即将发布的 Exchangis1.0，将与 DSS 工作流打通 | 不支持 | >= Exchangis1.0.0（**开发中**） |
+| [**Visualis**](https://github.com/WeBankFinTech/Visualis)   | 基于宜信开源项目 Davinci 二次开发的数据可视化 BI 工具，为用户在数据安全方面提供金融级数据可视化能力。 | = Visualis0.5.0 | >= Visualis1.0.0（**开发中**） |
+| [**Prophecis**](https://github.com/WeBankFinTech/Prophecis)     | 一站式机器学习平台，集成多种开源机器学习框架。Prophecis 的 MLFlow 通过 AppConn 可以接入到 DSS 工作流中。      | 不支持 | >= Prophecis 0.3.0（**开发中**） |
+| **UserManager** | （DSS 已内置的第三方应用工具）自动初始化一个 DSS 新用户所必须的所有用户环境，包含：创建 Linux 用户、各种用户路径、目录授权等。                 |  >= DSS0.9.1（**已发布**） | 规划在 DSS1.0.2（**开发中**） |
+| [**DolphinScheduler**](https://github.com/apache/dolphinscheduler) | Apache DolphinScheduler，分布式易扩展的可视化工作流任务调度平台，支持一键将DSS工作流发布到 DolphinScheduler。    | 不支持 | >= DolphinScheduler1.3.6，规划在 DSS1.1.0（**开发中**） |
+| **UserGuide**     | （DSS 将内置的第三方应用工具）包含帮助文档、新手指引、Dark模式换肤等。      | 不支持 | 规划在 DSS1.1.0（**开发中**） |
+| **DataModelCenter**     | （DSS 将内置的第三方应用工具）主要提供数仓规划、数据模型开发和数据资产管理的能力。数仓规划包含主题域、数仓分层、修饰词等；数据模型开发包含指标、维度、度量、向导式建表等；数据资产打通 **Apache Atlas**，提供**数据血缘**能力。      | 不支持 | 规划在 DSS1.2.0（**开发中**） |
+| [**Airflow**](https://github.com/apache/airflow)     | 支持将 DSS 工作流发布到 Apache Airflow 进行定时调度。      | <= DSS0.9.1，尚未合并 | 不支持 |
 
 
 ## Demo试用环境

diff --git a/README.md b/README.md
@@ -77,23 +77,24 @@ English | [中文](README-ZH.md)
 
  &nbsp; &nbsp; &nbsp; &nbsp;**If desired, new data application systems can also be easily integrated to replace or enrich DSS's data application development process.** [Click me to learn how to quickly integrate new application systems](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/en_US/Development_Documentation/Third-party_System_Access_Development_Guide.md)
 
-| Component | Description | DSS0.X Version Requirements | DSS1.0 Version Requirements | Version Planning |
-| --------------- | -------------------------------------------------------------------- | --------- | ---------- | ------ |
-| [**DataApiService**](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/en_US/Using_Document/DataApiService_Usage_Documentation.md) | Data API service. The SQL script can be quickly published as a Restful interface, providing Rest access capability to the outside world. | Not supported | >=1.0.0 | Released |
-| [**Scriptis**](https://github.com/WeBankFinTech/Scriptis) | Support online script writing such as SQL, Pyspark, HiveQL, etc., submit to [Linkis](https://github.com/WeBankFinTech/Linkis ) to perform data analysis web tools. | >=0.5.0 | >=1.0.0 | Released |
-| [**Schedulis**](https://github.com/WeBankFinTech/Schedulis) | Workflow task scheduling system based on Azkaban secondary development, with financial-grade features such as high performance, high availability and multi-tenant resource isolation. | >=0.5.0 | >=1.0.0 | Released |
-| **EventCheck** | Provides cross-business, cross-engineering, and cross-workflow signaling capabilities. | >=0.5.0 | >=1.0.0 | Released |
-| **SendEmail** | Provides the ability to send data, all the result sets of other workflow nodes can be sent by email | >=0.5.0 | >=1.0.0 | Released |
-| [**Qualitis**](https://github.com/WeBankFinTech/Qualitis) | Data quality verification tool, providing data verification capabilities such as data integrity and correctness | >=0.5.0 | 1.0.1(Version currently in preparation) | **Expected end of January** |
-| [**Streamis**](https://github.com/WeBankFinTech/Streamis) | Streaming application development management tool. It supports the release of Flink Jar and Flink SQL, and provides the development, debugging and production management capabilities of streaming applications, such as: start-stop, status monitoring, checkpoint, etc. | Not supported | 1.0.1(Version currently in preparation) | **Expected end of January** |
-| [**Exchangis**](https://github.com/WeBankFinTech/Exchangis) | A data exchange platform that supports data transmission between structured and unstructured heterogeneous data sources, the upcoming Exchangis1. 0, will be connected with DSS workflow | not supported | Planned in 1.0.2 | **In Development** |
-| [**Visualis**](https://github.com/WeBankFinTech/Visualis) | A data visualization BI tool based on the second development of Davinci, an open source project of CreditEase, provides users with financial-level data visualization capabilities in terms of data security. | >=0.5.0 | Planned in 1.0.2 | **In Development** |
-| [**Prophecis**](https://github.com/WeBankFinTech/Prophecis) | A one-stop machine learning platform that integrates multiple open source machine learning frameworks. Prophecis' MLFlow can be connected to DSS workflow through AppConn. | Not supported | Planned in 1.0.2 | **In Development** |
-| **UserManager** | Automatically initialize all user environments necessary for a new DSS user, including: creating Linux users, various user paths, directory authorization, etc. | >=0.9.1 | Planned in 1.0.2 | **In Development** |
-| **DolphinScheduler** | Apache DolphinScheduler, a distributed and scalable visual workflow task scheduling platform, supports one-click publishing of DSS workflows to DolphinScheduler. | Not supported | Planned in 1.1.0 | **In Development** |
-| **UserGuide**     | It mainly provides help documentation, beginner's guide, Dark mode skinning, etc.      | Not supported | Planning in 1.1.0 | **In Development** |
-| **DataModelCenter** | It mainly provides the capabilities of data warehouse planning, data model development and data asset management. Data warehouse planning includes subject domains, data warehouse layers, modifiers, etc.; data model development includes indicators, dimensions, metrics, wizard-based table building, etc.; data assets are connected to Apache Atlas to provide data lineage capabilities. | Not supported | Planning in 1.2.0 | **In Development** |
-| **Airflow** | Supports publishing DSS workflows to Airflow for scheduling. | >=0.9.1, not yet merged | Not supported | **No plans yet** |
+| Component | Description | DSS0.X Compatibility(recommend DSS0.9.1) | DSS1.0 Compatibility(recommend DSS1.0.1) |
+| --------------- | -------------------------------------------------------------------- | --------- | ---------- |
+| [**Linkis**](https://github.com/apache/incubator-linkis) | Apache Linkis, builds a layer of computation middleware, by using standard interfaces such as REST/WS/JDBC provided by Linkis, the upper applications can easily access the underlying engines such as MySQL/Spark/Hive/Presto/Flink, etc.  | >= Linkis0.5.0 (**Released**) | >= Linkis1.0.0 (**Released**)|
+| [**DataApiService**](https://github.com/WeBankFinTech/DataSphereStudio-Doc/blob/main/en_US/Using_Document/DataApiService_Usage_Documentation.md) | (Third-party applications built into DSS) Data API service. The SQL script can be quickly published as a Restful interface, providing Rest access capability to the outside world. | Not supported | >= DSS1.0.0 (**Released**) |
+| [**Scriptis**](https://github.com/WeBankFinTech/DataSphereStudio) | (Third-party applications built into DSS) Support online script writing such as SQL, Pyspark, HiveQL, etc., submit to [Linkis](https://github.com/WeBankFinTech/Linkis ) to perform data analysis web tools. | >= DSS0.5.0 (**Released**) | >= DSS1.0.0 (**Released**) |
+| [**Schedulis**](https://github.com/WeBankFinTech/Schedulis) | Workflow task scheduling system based on Azkaban secondary development, with financial-grade features such as high performance, high availability and multi-tenant resource isolation. | <= Schedulis0.6.1 (**Released**) | >= Schedulis0.6.2 (**Developing**) |
+| **EventCheck** | (Third-party applications built into DSS) Provides cross-business, cross-engineering, and cross-workflow signaling capabilities. | >= DSS0.5.0 (**Released**) | >= DSS1.0.0 (**Released**) |
+| **SendEmail** | (Third-party applications built into DSS) Provides the ability to send data, all the result sets of other workflow nodes can be sent by email | >=0.5.0 (**Released**) | >=1.0.0 (**Released**) |
+| [**Qualitis**](https://github.com/WeBankFinTech/Qualitis) | Data quality verification tool, providing data verification capabilities such as data integrity and correctness | >= Qualitis0.8.0 (**Released**) | >= Qualitis0.9.0 (**Developing**) |
+| [**Streamis**](https://github.com/WeBankFinTech/Streamis) | Streaming application development management tool. It supports the release of Flink Jar and Flink SQL, and provides the development, debugging and production management capabilities of streaming applications, such as: start-stop, status monitoring, checkpoint, etc. | Not supported | >= Streamis0.1.0 (**Developing**) |
+| [**Exchangis**](https://github.com/WeBankFinTech/Exchangis) | A data exchange platform that supports data transmission between structured and unstructured heterogeneous data sources, the upcoming Exchangis1. 0, will be connected with DSS workflow | not supported | >= Exchangis1.0.0 (**Developing**) |
+| [**Visualis**](https://github.com/WeBankFinTech/Visualis) | A data visualization BI tool based on the second development of Davinci, an open source project of CreditEase, provides users with financial-level data visualization capabilities in terms of data security. | = Visualis0.5.0 (**Released**) | >= Visualis1.0.0 (**Developing**) |
+| [**Prophecis**](https://github.com/WeBankFinTech/Prophecis) | A one-stop machine learning platform that integrates multiple open source machine learning frameworks. Prophecis' MLFlow can be connected to DSS workflow through AppConn. | Not supported | >= Prophecis 0.3.0 (**Developing**) |
+| **UserManager** | (Third-party applications built into DSS) Automatically initialize all user environments necessary for a new DSS user, including: creating Linux users, various user paths, directory authorization, etc. | >= DSS0.9.1 (**Released**) | Planned in DSS1.0.2 (**Developing**) |
+| [**DolphinScheduler**](https://github.com/apache/dolphinscheduler) | Apache DolphinScheduler, a distributed and scalable visual workflow task scheduling platform, supports one-click publishing of DSS workflows to DolphinScheduler. | Not supported | >= DolphinScheduler1.3.6, planned in DSS1.1.0 (**Developing**) |
+| **UserGuide**     | (Third-party applications to be built into DSS) It mainly provides help documentation, beginner's guide, Dark mode skinning, etc.      | Not supported | Planning in DSS1.1.0 (**Developing**) |
+| **DataModelCenter** | (Third-party applications to be built into DSS) It mainly provides the capabilities of data warehouse planning, data model development and data asset management. Data warehouse planning includes subject domains, data warehouse layers, modifiers, etc.; data model development includes indicators, dimensions, metrics, wizard-based table building, etc.; data assets are connected to Apache Atlas to provide data lineage capabilities. | Not supported | Planning in DSS1.2.0 (**Developing**) |
+| [**Airflow**](https://github.com/apache/airflow) | Supports publishing DSS workflows to Apache Airflow for scheduling. | >= DSS0.9.1, not yet merged | Not supported |
 
 
 ## Demo Trial environment

diff --git a/assembly/dss-package/pom.xml b/assembly/dss-package/pom.xml
@@ -127,6 +127,12 @@
             <version>2.21</version>
         </dependency>
 
+        <dependency>
+            <groupId>mysql</groupId>
+            <artifactId>mysql-connector-java</artifactId>
+            <version>${mysql.version}</version>
+        </dependency>
+
     </dependencies>
 
     <build>

diff --git a/dss-appconn/linkis-appconn-engineplugin/src/main/resources/linkis-engineconn.properties b/dss-appconn/linkis-appconn-engineplugin/src/main/resources/linkis-engineconn.properties
@@ -47,4 +47,7 @@ wds.linkis.mysql.is.encrypt=false
 wds.linkis.reflect.scan.package=org.apache.linkis,com.webank.wedatasphere.dss
 spring.spring.mvc.servlet.path=/api/rest_j/v1
 spring.spring.servlet.multipart.max-file-size=200MB
-spring.spring.servlet.multipart.max-request-size=200MB
+spring.spring.servlet.multipart.max-request-size=200MB
+
+wds.linkis.engineconn.support.parallelism=true
+wds.linkis.engineconn.max.free.time=0
diff --git a/.../org/apache/linkis/manager/engineplugin/appconn/conf/AppConnEngineConnConfiguration.scala b/.../org/apache/linkis/manager/engineplugin/appconn/conf/AppConnEngineConnConfiguration.scala
@@ -23,4 +23,6 @@ object AppConnEngineConnConfiguration {
 
   val GATEWAY_SPRING_APPLICATION = CommonVars("wds.linkis.gateway.spring.name", "dataworkcloud-gateway")
 
+  val CONCURRENT_LIMIT = CommonVars("wds.linkis.engineconn.appconn.conncurrent.limit", 100)
+
 }