HPCC-24058 Add manifest command to esdl tool #16120

asselitx · 2022-05-23T20:17:28Z

The manifest command builds an ESDL service binding or bundle from an input
manifest file. The manifest contains the outline of the binding and can use
Include statements to load external files to include in the output. Supported
Include files are ESDL scripts, XSL transforms for logging and ESDL
definitions.

Add tests to to exercise the esdl manifest command.
Refactor the EsdlExample into separate independent examples, including one
showing the use of the manifest tool.
Add a README explaining the use of the new command.

The bulk of the changes in this PR are related to updates to the EsdlExample.
So although there are many changed/deleted/moved files, the code changes
for implementing the new command are within the esdl tool project folder.

Signed-off-by: Terrence Asselin terrence.asselin@lexisnexisrisk.com

Type of change:

This change is a bug fix (non-breaking change which fixes an issue).
This change is a new feature (non-breaking change which adds functionality).
This change improves the code (refactor or other change that does not change the functionality)
This change fixes warnings (the fix does not alter the functionality or the generated code)
This change is a breaking change (fix or feature that will cause existing behavior to change).
This change alters the query API (existing queries will have to be recompiled)

Checklist:

Smoketest:

Send notifications about my Pull Request position in Smoketest queue.
Test my draft Pull Request.

Testing:

Added new regression cases in the testing/esp/esdlcmd project.

github-actions · 2022-05-23T20:18:03Z

https://track.hpccsystems.com/browse/HPCC-24058
Jira updated

afishbeck · 2022-05-26T21:02:08Z

I think initfiles/examples/EsdlExample/Manifest needs a copy of RoxieEchoPersonInfo.ecl for the walkthrough to work.

afishbeck · 2022-05-26T21:41:52Z

initfiles/examples/EsdlExample/CppPlugin/README.md

+
+9. Test calling the roxie service
+```
+soapplus -url http://.:8088/EsdlExample -i cpprequest.xml


like above, should be: "../../cpprequest.xml"

afishbeck · 2022-05-27T19:59:54Z

tools/esdlcmd/esdlcmd_common.hpp

@@ -142,6 +142,13 @@ typedef IEsdlCommand *(*EsdlCommandFactory)(const char *cmdname);
 #define ESDLOPT_INCLUDE_PATH_ENV        "ESDL_INCLUDE_PATH"
 #define ESDLOPT_INCLUDE_PATH_INI        "esdlIncludePath"
 #define ESDLOPT_INCLUDE_PATH_USAGE      "   -I, --include-path <include path>    Locations to look for included esdl files\n"
+#define ESDLOPT_RECURSIVE               "--recursive"
+#define ESDLOPT_RECURSIVE_S             "--r"


Short names have one dash, long names have two. So this should be simply "-r".

afishbeck · 2022-05-27T20:10:10Z

tools/esdlcmd/esdlcmd_core.cpp

+        puts("Options:");
+        puts("   --rollup         Output a single XML file containing all definitions used by");
+        puts("                    <serviceName>");
+        puts("   -r, --recursive  Process all includes. Implied when generating monolithic");


I see the help is correct for the short version with a single dash. The define must have been a typo.

Yes I think so.

afishbeck · 2022-05-27T20:11:08Z

tools/esdlcmd/esdlcmd_manifest.cpp

@@ -0,0 +1,884 @@
+/*##############################################################################
+
+    HPCC SYSTEMS software Copyright (C) 2020 HPCC Systems®.


timothyklemm · 2022-05-24T11:24:01Z

initfiles/examples/EsdlExample/BasicRoxie/README.md

@@ -0,0 +1,66 @@
+```
+    HPCC SYSTEMS software Copyright (C) 2015 HPCC Systems®.


New files need more recent copyright notices.

timothyklemm · 2022-05-24T11:45:05Z

initfiles/examples/EsdlExample/BasicRoxie/README.md

+
+## DynamicESDL
+
+You must have configured at least one DynamicESDL ESP Service in this environment.  Either through configmgr or by editing environment.xml. The instance can be bound on port 0, or on the same port as where you're planning to run your service, or both. The purpose of the instance is to provide the configuration needed for your service. If there's an instance configured on the same port, that instance's configuration will be used, otherwise the configuration of the instance on port 0 will be used. This document assumes you're planning to bind your service to esp process "myesp" and port 8088.


Probably too much emphasis on the old way of setting things up, and no mention of working with containers.

I agree that some newer examples including containers will be good, but I'd prefer not to include that in the scope of this ticket. When adding my manifest example to the directory it was difficult to understand how the examples worked, they were intermingled and they didn't have a consistent structure. I worried that just putting a new example in the mix would be unhelpful.

So I wanted to make them easier to run and understand without changing their basic nature. I will be explicit in the README that these are examples of bare-metal deployments, which I think is useful to have documented. If you'd like I'll also open a ticket to add container and/or esp app-based instructions as either separate examples or updates to these.

timothyklemm · 2022-05-24T11:49:09Z

initfiles/examples/EsdlExample/BasicRoxie/README.md

+
+## EclWatch
+
+An instance of EclWatch must be running, and you must have shell access to that node.


Why? Needing a running instance is understandable. Knowing that some commands accept options to redirect to remote instances, is the requirement to be local because a specific command does not support this or is it convenience?

Good point- trying to be too general is maybe just confusing here. I'll update the readmes to all use the assumption that you are running the stock-installed environment.xml on a single machine that you can access.

timothyklemm · 2022-05-24T11:55:41Z

initfiles/examples/EsdlExample/Manifest/README.md

+
+4. Create the binding from a manifest file
+```
+esdl manifest manifest_roxie.xml -I EsdlScripts --outfile manifest_example_binding.xml


manifest_roxie.xml seems like an odd choice of name for a file used to define an ESP binding.

All the examples are named in terms of how the 'backend' is implemented- I guess I was just running on that track. I'll rename it not referencing roxie.

timothyklemm · 2022-05-24T12:00:50Z

initfiles/examples/EsdlExample/Manifest/README.md

+esdl bind-service myesp 8088 esdlexample.1 EsdlExample --config manifest_example_binding.xml --overwrite
+```
+
+1. Test calling the roxie service


Number is wrong. Markdown viewers might fix, but it's out of place when all of the other numbers are sequential.

Is this testing the roxie or the ESP? If the ESP is on 8088, it seems like it's an ESP test. The roxie might be tested indirectly, but that's a function of the ESP.

Fixed numbering and adjusted wording, though in keeping with the idea that these examples largely differ by what the backend or implementation is for each.

timothyklemm · 2022-06-01T13:51:25Z

tools/esdlcmd/esdlcmd_manifest.cpp

+                if (!content.length())
+                    throw makeStringExceptionV(ESDL_MANIFEST_ERR, "Unable to load <Include>, file='%s' on line #%d column #%d", filename, xpp->getLineNumber(), xpp->getColumnNumber());
+            }
+            else if (INCLUDE_LOG_TRANSFORM == includeType)


Not a fan of the named symbol. Perhaps INCLUDE_XSLT would be better, since we may end up supporting XSLTs that are not specific to logging.

A name for this case is required as long as arbitrary includes are not supported. If arbitrary includes are supported, a name for this case is not required unless it is used to control handling of the file content (e.g., whitespace preservation instead of pretty-printing).

I've updated the symbol name as suggested. Future update for arbitrary includes may include a refactoring of this method, so I'll address possible removal of the symbol all together at that time.

timothyklemm · 2022-06-01T13:56:36Z

tools/esdlcmd/esdlcmd_manifest.cpp

+
+        void appendStartTag(IXmlPullParser* xpp, StartTag& stag, bool writeNsAttr = false)
+        {
+            // Attributes don't have their ns prefixes copied


I don't read this comment as I think it is intended. Based on the code, I think you're saying that namespace declarations are not included in stag, so you are manually adding them.

timothyklemm · 2022-06-01T13:58:16Z

tools/esdlcmd/esdlcmd_manifest.cpp

+                while (it != xpp->getNsEnd())
+                {
+                    if (it->first.compare("xml")!=0)
+                        binding.appendf(" xmlns:%s=\"%s\"", it->first.c_str(), it->second);


Does this handle the default namespace, which should be xmlns=... instead of xmlns:=...?

It appears that the xpp namespace iterator does not include the default namespace. That can only be inferred by interrogating the uri of a start tag. To see if the default namespace was defined, you must compare a start tag's uri to a list of already seen uris. If it is new then this node is the first to use the default, though it may have been defined in a parent. There is no way to determine which node defined the default. Updated to handle default namespaces given these constraints.

timothyklemm · 2022-06-01T14:10:28Z

tools/esdlcmd/esdlcmd_manifest.cpp

+            return false;
+        }
+
+        void appendStartTag(IXmlPullParser* xpp, StartTag& stag, bool writeNsAttr = false)


Default parameters are frowned upon.

Are we artificially limiting users by conditionally handling namespace declarations? I would not want to see every start tag redeclare every namespace, but it shouldn't be too difficult to identify when a start tag has declared a new namespace by maintaining some state locally.

Removed default parameter and generalized handling of namespaces.

timothyklemm · 2022-06-01T14:29:23Z

tools/esdlcmd/esdlcmd_manifest.cpp

+                    case XmlPullParser::END_DOCUMENT:
+                        level = 0;
+                        break;


This seems logically incorrect. If this case can be reached, the XML is invalid and we should object. If the case cannot be reached, because the parser throws an exception, it is extraneous and misleading.

Changed all occurrences except at the root level to throw an exception.

ghalliday · 2022-09-23T10:35:14Z

@asselitx this looks like it has stalled. Please can you either close, or respond to the comments.

asselitx · 2022-09-26T16:16:18Z

@ghalliday I will resume work on this PR later this week.

HPCCSmoketest · 2022-11-03T12:51:56Z

This PR (16120) is in the 11/56 position of the queue. Estimated start time is after ~5.10 hour(s)

ghalliday · 2022-11-11T10:45:51Z

@asselitx still stalled...

asselitx · 2022-12-30T21:11:59Z

@afishbeck and @timothyklemm could you please review the updates? I didn't rebase to latest or squash yet to help keep the diffs clean. The follow-up ticket for requested changed not addressed now is HPCC-28740.

timothyklemm

This review only covers esdlcmd_manifest.cpp.

timothyklemm · 2023-01-03T19:34:47Z

tools/esdlcmd/esdlcmd_manifest.cpp

+
+        EsdlManifestCmd() {}
+
+        int processCMD() override


missing virtual keyword

Is this style? I thought an override of a virtual method was implicitly virtual in all derived classes where its overridden. Core cpp guidelines suggest to use one or the other: http://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines.html#c128-virtual-functions-should-specify-exactly-one-of-virtual-override-or-final

fixed to conform to HPCC guidelines

timothyklemm · 2023-01-03T19:35:32Z

tools/esdlcmd/esdlcmd_manifest.cpp

+        EsdlManifestCmd() {}
+
+        int processCMD() override
+        {   bool result = false;


missing newline?

tools/esdlcmd/esdlcmd_manifest.cpp

timothyklemm · 2023-01-03T19:41:09Z

tools/esdlcmd/esdlcmd_manifest.cpp

+            if (iter.done())
+            {
+                usage();
+            } else {


inconsistent formatting

timothyklemm · 2023-01-03T19:55:46Z

tools/esdlcmd/esdlcmd_manifest.cpp

+            puts("    --output-format <format>");
+            puts("                        Specify the format of the output. Allowed values are");
+            puts("                        'binding' or 'bundle'. The default is 'bundle' which");
+            puts("                        includes the ESDL Service Definition (if provided). This");
+            puts("                        option overrides Manifest[@outputFormat] in the file.");


I probably commented on this before, but this still bothers me. I read it as Manifest[@outputFormat] is always irrelevant. A default of bundle should only be applied if neither the manifest itself nor the command line specifies something different.

Tried to clear up the language.

timothyklemm · 2023-01-04T15:13:01Z

tools/esdlcmd/esdlcmd_manifest.cpp

+         * @param root reference to the 'ServiceDefinition' node read
+         * @return int type of the last node read
+         */
+        int processServiceDefinition(IXmlPullParser* xpp, StartTag& root)


Can this be renamed to something like processEsdlServiceDefinition?

sure- or perhaps EsdlDefinition? Either way if I change it should I change the element name to match?

If a bundle is viewed as a superset of a binding, maybe processBundleDefinitions versus processBindingDefinition. I wouldn't complain about consistency between the markup and function name.

Tony said he'd prefer EsdlDefinition, so I'll change that name for the function and for the manifest namespace element.

tools/esdlcmd/esdlcmd_manifest.cpp

timothyklemm · 2023-01-04T15:32:33Z

tools/esdlcmd/esdlcmd_manifest.cpp

+                        {
+                            bindingFileIO->write(0, binding.length(), binding.str());
+                        } else {
+                            UERRLOG("Error writing to binding file (%s).", optOutputPath.str());


Ignoring the potential of other artifact types, not all output files are binding files. Better to keep it generic and omit "binding".

timothyklemm · 2023-01-04T15:32:50Z

tools/esdlcmd/esdlcmd_manifest.cpp

+            }
+            return result;
+        }
+};


timothyklemm

Comments on tools/esdlcmd/README.md only.

timothyklemm · 2023-02-17T14:53:31Z

tools/esdlcmd/README.md

+
+### Manifest File
+
+A manifest file is an XML-formatted template combining elements in and outside of the manifest's `urn:hpcc:esdl:manifest` namespace. Elements of this namespace control the tool while all other markup is copied to the output. The goal of using a manifest file with the tool is to make configuring and deploying services easier:


The second sentence is misleading assuming a previously stated intention to pass unrecognized manifest namespace markup to the output. I read this sentence as "no manifest elements are copied to the output".

timothyklemm · 2023-02-17T14:55:20Z

tools/esdlcmd/README.md

+    </em:EsdlDefinition>
+</em:Manifest>
+```
+Because the tool is permissive and flexible, copying through all markup not in the manifest namespace, the manifest elements are only required in order to take advantage of the automated processing and simplified format of the manifest file. This example highlights the recommended usage of manifest elements to use the tool's capabilities. Although you could replace some of the elements below with verbatim `bundle` or `binding` output elements we won't cover that usage here.


Sentence one again suggests no manifest namespace markup will be copied to the output.

timothyklemm · 2023-02-17T14:57:04Z

tools/esdlcmd/README.md

+```
+Because the tool is permissive and flexible, copying through all markup not in the manifest namespace, the manifest elements are only required in order to take advantage of the automated processing and simplified format of the manifest file. This example highlights the recommended usage of manifest elements to use the tool's capabilities. Although you could replace some of the elements below with verbatim `bundle` or `binding` output elements we won't cover that usage here.
+
+* `<em:Manifest>` Is the required root element. By default the tool outputs a `bundle`, though you may explicitly override that on the command line or by providing an `@outputType=binding` attribute.


Should Is be capitalized? It doesn't match the rest of the list.

timothyklemm · 2023-02-17T14:59:56Z

tools/esdlcmd/README.md

+```
+Because the tool is permissive and flexible, copying through all markup not in the manifest namespace, the manifest elements are only required in order to take advantage of the automated processing and simplified format of the manifest file. This example highlights the recommended usage of manifest elements to use the tool's capabilities. Although you could replace some of the elements below with verbatim `bundle` or `binding` output elements we won't cover that usage here.
+
+* `<em:Manifest>` Is the required root element. By default the tool outputs a `bundle`, though you may explicitly override that on the command line or by providing an `@outputType=binding` attribute.


@outputType=binding is incorrect when presented as markup. binding should be quoted for the markup to be correct.

timothyklemm · 2023-02-17T15:00:24Z

tools/esdlcmd/README.md

+Because the tool is permissive and flexible, copying through all markup not in the manifest namespace, the manifest elements are only required in order to take advantage of the automated processing and simplified format of the manifest file. This example highlights the recommended usage of manifest elements to use the tool's capabilities. Although you could replace some of the elements below with verbatim `bundle` or `binding` output elements we won't cover that usage here.
+
+* `<em:Manifest>` Is the required root element. By default the tool outputs a `bundle`, though you may explicitly override that on the command line or by providing an `@outputType=binding` attribute.
+* `em:ServiceBinding>` is valid for both `bundle` and `binding` output. It is necessary to enable recognition of `<em:Scripts>`, and `<em:Transform>` elements.


Missing opening <.

timothyklemm · 2023-02-17T15:24:53Z

tools/esdlcmd/README.md

+
+| Attribute   | Required? | Value  | Usage |
+| - | :-: | - | - |
+| `@auth_feature` | No | string | Used declare authorization settings if they aren't present in the ESDL Definition, or override them if they are. Additional documentation on this attribute is forthcoming. |


"Used to declare ..."

timothyklemm · 2023-02-17T15:27:27Z

tools/esdlcmd/README.md

+The element is not required since it is possible to embed a complete `Definitions` element hierarchy in the manifest.
+
+#### Include
+Optional element that imports the contents of another file into the output, in place of the itself. The outcome of the import depends on the context in which this element is used. See [EsdlDefinition](#esdldefinition), [Scripts](@scripts), and [Transform](#transform) for more information.


"... in place of the itself." Remove "the"

timothyklemm · 2023-02-17T15:28:33Z

tools/esdlcmd/README.md

+| - | :-: | - | - |
+| `@file` | Yes | file path | Full or partial path to an external file to be imported. If a partial path is outside of the tool's working directory, the tool's command line must specify the appropriate root directory using either `-I` or `--include-path`. |
+
+>Any XSLTs or ESDL Scripts written inline in a manifest file will have XML escaping applied where required to generate valid XML. If an XLST contains any text content or markup that needs to be preserved as-is (no XML escaping applied) then be sure to use an `<em:Include>` operation. Included files are inserted into the output as-is, with the exception of preventing nested CDATA. If the included file will be inside a CDATA section on output, then any CDATA end markup in the file will be encoded as `]]]]><![CDATA[>` to prevent nested CDATA sections or a prematurely ending a CDATA section.


XLST -> XSLT

timothyklemm · 2023-02-17T15:33:15Z

tools/esdlcmd/README.md

+Optional repeatable element appearing within an `<em:ServiceBinding>` element that processes child elements and creates output expected for an ESDL binding:
+
+- The `<em:Scripts>` element is replaced on output with `<Scripts>`. Then all content is enclosed in a CDATA section after wrapping it with a new `Scripts` element. That new `<Scripts>` element contains namespaces declared by the input `<em:Scripts>` element. The input `<em:Scripts foo="..." xmlns:bar="..."><!-- content --></em:Scripts>` becomes `<Scripts><![CDATA[<Scripts xmlns:bar="..."><!-- content --></Scripts>]]></Scripts>`.
+- The manifest `<em:Include>` element is recognized to import scripts from external files. The entire file, minus leading and trailing whitespace, is imported.


If sentence two is correct, perhaps a warning against including files that include XML declarations.

timothyklemm · 2023-02-17T15:33:31Z

tools/esdlcmd/README.md

+Optional repeatable element appearing within an `<em:ServiceBinding>` element that processes child elements and creates output expected for an ESDL binding:
+
+- All content is enclosed in a CDATA section. The input `<em:Transform> <!-- content --> </em:Transform>` becomes `<Transform><![CDATA[<!-- content -->]]></Transform>`.
+- The manifest `<em:Include>` element is recognized to import transforms from external files. The entire file, minus leading and trailing whitespace, is imported.


timothyklemm

Just esdlcmd_manifest.cpp ...

tools/esdlcmd/esdlcmd_manifest.cpp

timothyklemm · 2023-02-17T16:39:25Z

tools/esdlcmd/esdlcmd_manifest.cpp

+            if (isAbsolutePath(filePath))
+                return false;
+
+            for (std::string include : includes)


Use reference to avoid the copy.

timothyklemm · 2023-02-17T16:41:02Z

tools/esdlcmd/esdlcmd_manifest.cpp

+            }
+            while(type != XmlPullParser::END_TAG && type != XmlPullParser::END_DOCUMENT);
+            EndTag etag;
+            xpp->readEndTag(etag);


What happens if we exited the loop with END_DOCUMENT?

Probably some undefined behavior. Fixed.

tools/esdlcmd/esdlcmd_manifest.cpp

timothyklemm · 2023-02-17T19:41:13Z

tools/esdlcmd/esdlcmd_manifest.cpp

+                    case XmlPullParser::END_TAG:
+                        xpp->readEndTag(etag);
+
+                        if (isRootEndTagNamed(etag, "Scripts", level))


I still see "Scripts".

tools/esdlcmd/esdlcmd_manifest.cpp

afishbeck · 2023-05-24T21:09:05Z

tools/esdlcmd/esdlcmd_manifest.cpp

@@ -0,0 +1,1310 @@
+/*##############################################################################
+
+    HPCC SYSTEMS software Copyright (C) 2022 HPCC Systems®.


It's now 2023, I see a comment from a year ago, was this draft around that long?

Fixed in today's commit.

afishbeck · 2023-05-24T21:43:39Z

tools/esdlcmd/esdlcmd_manifest.cpp

+                    if (ManifestType::bundle == outputType)
+                        appendEndTag(MANIFEST_TAG_BUNDLE);
+                    if (ManifestType::bundle == outputType && !foundEsdlDefn)
+                        UWARNLOG("Generating EsdlBundle output without ESDL Definition. This may cause runtime issues.");


Perhaps this should be an error? I think all bundles should have interfaces defined? If in the future we wanted to make it a warning we could. Sometimes starting out strict is good.

I'm fine with that. I'll plan to throw an exception to prevent the creation of a bundle without the interface- unless you meant just to use UERRLOG?

Throw an exception, we can always relax it later if it is really needed.

Fixed in today's commit.

asselitx · 2023-07-13T19:52:55Z

@timothyklemm and @afishbeck I believe I've addressed all outstanding comments. Please check latest commit. I'll rebase and squash onto 9.2.x when you think it looks good.

asselitx · 2023-07-21T16:32:05Z

@timothyklemm and @afishbeck I've rebased to candidate-9.2.x and fixed the README error in grammar I'd missed. I feel the changes are safe in 9.2 because they're limited to

The initfiles/examples
esdl command tests
new manifest esdl command
adding support for esdl xml to run directly from the esdl app instead of a separate executable

timothyklemm · 2023-08-03T14:42:49Z

tools/esdlcmd/README.md


 | Attribute   | Required? | Value  | Usage |
 | - | :-: | - | - |
 | `@auth_feature` | No | string | Used declare authorization settings if they aren't present in the ESDL Definition, or override them if they are. Additional documentation on this attribute is forthcoming. |
+| `@returnSchemaLocationOnOK` | No | Boolean | When true, a successful SOAP response (non SOAP-Fault) will include the schema location property. False by default. |
+| `@namespace` | No | string | String specifying the namespace for all methods in the binding. May contain variables that are replaced with their values: <br/>- `${service}` : lowercase service name <br/>- `${esdl-service}` : service name, possibly mixed case <br/>- `${method}` : lowercase method name <br/>- `${esdl-method}` : method name, possibly mixed case <br/>- `${optionals}` : comma-delimited list of all optional URL parameters included in the method request, enclosed in parenthesis <br/>- `${version}` : client version number |


Should "parenthesis" be "parentheses" (plural instead of singular)?

makes sense, fixed

timothyklemm · 2023-08-03T14:51:09Z

tools/esdlcmd/README.md

+| - | :-: | - | - |
+| `@auth_feature` | No | string | Used to declare authorization settings if they aren't present in the ESDL Definition, or override them if they are. Additional documentation on this attribute is forthcoming. |
+| `@returnSchemaLocationOnOK` | No | Boolean | When true, a successful SOAP response (non SOAP-Fault) will include the schema location property. False by default. |
+| `@namespace` | No | string | String specifying the namespace for all methods in the binding. May contain variables that are replaced with their values: <br/>- `${service}` : lowercase service name <br/>- `${esdl-service}` : service name, possibly mixed case <br/>- `${method}` : lowercase method name <br/>- `${esdl-method}` : method name, possibly mixed case <br/>- `${optionals}` : comma-delimited list of all optional URL parameters included in the method request, enclosed in parenthesis <br/>- `${version}` : client version number |


When does the variable replacement happen? I couldn't fault someone for expecting the manifest tool to replace the values. I also couldn't fault someone for thinking the variables are retained for the ESP to replace.

Clarified the ESP replaces them during runtime.

timothyklemm · 2023-08-22T14:44:48Z

tools/esdlcmd/README.md

+| `@port`        | No | string | Port on the ESP Process listening for connections to the binding. |
+| `@publishedBy` | No | string | Userid of the person publishing the binding |
+
+The `<em:ServiceBinding>` element is not required since it is possible to embed a complete `Binding` element hierarchy in the manifest.


Consider moving before line 122. It clarifies line 115's reference to "recommended", a meaning which might be lost this far away.

Yeah seems like that got shuffled during edits. I've reworded slightly and moved up.

timothyklemm · 2023-08-22T14:49:18Z

tools/esdlcmd/README.md

+| - | :-: | - | - |
+| `@file` | Yes | file path | Full or partial path to an external file to be imported. If a partial path is outside of the tool's working directory, the tool's command line must specify the appropriate root directory using either `-I` or `--include-path`. |
+
+>Any XSLTs or ESDL Scripts written inline in a manifest file will have XML escaping applied where required to generate valid XML. If an XSLT contains any text content or markup that needs to be preserved as-is (no XML escaping applied) then be sure to use an `<em:Include>` operation. Included files are inserted into the output as-is, with the exception of preventing nested CDATA. If the included file will be inside a CDATA section on output, then any CDATA end markup in the file will be encoded as `]]]]><![CDATA[>` to prevent nested CDATA sections or a prematurely ending a CDATA section.


"preventing nested CDATA" seems wrong. The action taken properly encodes nested CDATA rather than preventing it.

timothyklemm · 2023-08-24T17:08:53Z

tools/esdlcmd/README.md

+
+#### Bundle
+
+The bundle includes all the components of the service binding in addition to the ESDL definition.


Consider rephrasing. Between this and 426, seems somewhat circular.

Removed that wording and instead called back to the usage for binding/bundle around line 45.

asselitx · 2023-12-12T16:08:35Z

@ghalliday approved for merge and squashed

The manifest command builds an ESDL service binding or bundle from an input manifest file. The manifest contains the outline of the binding and can use 'Include' statements to load external files to include in the output. Supported Include files are ESDL scripts, XSL transforms for logging and ESDL definitions. - Add tests to to exercise the esdl manifest command. - Refactor the EsdlExample into separate independent examples, including one showing the use of the manifest tool. - Add a README explaining the use of the new command. Signed-off-by: Terrence Asselin <terrence.asselin@lexisnexisrisk.com>

asselitx · 2024-01-04T22:08:58Z

Rebased to target branch again in an attempt to re-trigger the checks that had stalled out.

asselitx · 2024-01-17T17:24:40Z

@ghalliday this should be ready now for your final approval and merge. Let me know if I need to do anything else.

asselitx requested review from afishbeck and timothyklemm May 23, 2022 20:17

asselitx assigned timothyklemm May 23, 2022

afishbeck reviewed May 26, 2022

View reviewed changes

afishbeck reviewed May 27, 2022

View reviewed changes

timothyklemm requested changes Jun 1, 2022

View reviewed changes

asselitx marked this pull request as draft November 14, 2022 23:52

asselitx force-pushed the manifest-hpcc-24058 branch from a43f6a7 to 0c9afa5 Compare December 30, 2022 21:09

timothyklemm requested changes Jan 4, 2023

View reviewed changes

asselitx force-pushed the manifest-hpcc-24058 branch from 0c9afa5 to 0a572f9 Compare February 14, 2023 00:07

asselitx changed the base branch from candidate-8.6.x to candidate-8.12.x February 14, 2023 15:15

timothyklemm requested changes Feb 17, 2023

View reviewed changes

asselitx force-pushed the manifest-hpcc-24058 branch from 0a572f9 to 2098b0d Compare April 24, 2023 18:19

asselitx changed the base branch from candidate-8.12.x to candidate-9.0.x April 24, 2023 18:37

afishbeck reviewed May 24, 2023

View reviewed changes

asselitx assigned afishbeck Jul 13, 2023

asselitx marked this pull request as ready for review July 13, 2023 19:50

asselitx force-pushed the manifest-hpcc-24058 branch from da16522 to 0908aa0 Compare July 21, 2023 16:27

asselitx changed the base branch from candidate-9.0.x to candidate-9.2.x July 21, 2023 16:27

asselitx requested review from afishbeck and timothyklemm August 17, 2023 20:19

timothyklemm reviewed Aug 24, 2023

View reviewed changes

asselitx force-pushed the manifest-hpcc-24058 branch from 0908aa0 to 2f09b52 Compare August 30, 2023 15:13

asselitx requested a review from timothyklemm August 30, 2023 15:13

timothyklemm approved these changes Sep 1, 2023

View reviewed changes

afishbeck approved these changes Dec 12, 2023

View reviewed changes

asselitx force-pushed the manifest-hpcc-24058 branch from 2f09b52 to c1c5231 Compare December 12, 2023 16:06

asselitx changed the base branch from candidate-9.2.x to candidate-9.4.x December 12, 2023 16:07

asselitx assigned ghalliday and unassigned afishbeck and timothyklemm Dec 12, 2023

asselitx force-pushed the manifest-hpcc-24058 branch from c1c5231 to 6731d8c Compare January 4, 2024 22:07

asselitx requested a review from ghalliday January 17, 2024 17:22

ghalliday merged commit 8e68842 into hpcc-systems:candidate-9.4.x Jan 25, 2024

		@@ -0,0 +1,884 @@
		/*##############################################################################

		HPCC SYSTEMS software Copyright (C) 2020 HPCC Systems®.

		@@ -0,0 +1,66 @@
		```
		HPCC SYSTEMS software Copyright (C) 2015 HPCC Systems®.


		## DynamicESDL

		You must have configured at least one DynamicESDL ESP Service in this environment. Either through configmgr or by editing environment.xml. The instance can be bound on port 0, or on the same port as where you're planning to run your service, or both. The purpose of the instance is to provide the configuration needed for your service. If there's an instance configured on the same port, that instance's configuration will be used, otherwise the configuration of the instance on port 0 will be used. This document assumes you're planning to bind your service to esp process "myesp" and port 8088.


		## EclWatch

		An instance of EclWatch must be running, and you must have shell access to that node.


		### Manifest File

		A manifest file is an XML-formatted template combining elements in and outside of the manifest's `urn:hpcc:esdl:manifest` namespace. Elements of this namespace control the tool while all other markup is copied to the output. The goal of using a manifest file with the tool is to make configuring and deploying services easier:

		@@ -0,0 +1,1310 @@
		/*##############################################################################

		HPCC SYSTEMS software Copyright (C) 2022 HPCC Systems®.


		#### Bundle

		The bundle includes all the components of the service binding in addition to the ESDL definition.

HPCC-24058 Add manifest command to esdl tool #16120

HPCC-24058 Add manifest command to esdl tool #16120

Conversation

asselitx commented May 23, 2022 • edited Loading

Type of change:

Checklist:

Smoketest:

Testing:

github-actions bot commented May 23, 2022

afishbeck commented May 26, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asselitx Oct 13, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghalliday commented Sep 23, 2022

asselitx commented Sep 26, 2022

HPCCSmoketest commented Nov 3, 2022

ghalliday commented Nov 11, 2022

asselitx commented Dec 30, 2022

timothyklemm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothyklemm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asselitx commented May 23, 2022 •

edited

Loading

asselitx Oct 13, 2022 •

edited

Loading