Array/list inputs to engine and/or simulator?

[michaelglasson]

Hi again, Radek. I have reviewed the documentation and some of the source code, but I cannot confirm whether or not array/list inputs are usable either in the engine or the simulator. I am able to evaluate inputs like ["a","b","c"] in the Camunda engine using FEEL's 'some x in y satisfies z' construct.

I see that the simulator documentation says: When the type is not identified by parser from the definition (or it's identified but not as the known type), the entry is via text field and the combo box is provided to let the user choose a know type to which the raw value should be converted before passing to the execution context as the input parameter. I am guessing, however, that this does not allow me to enter an object like an array.

I think I can work around this by breaking my decision into two or more tables or expressions, so it's not a killer issue for me at the moment.

Cheers,
Michael

PS. It's really generous of you to contribute products like this to the public domain. Much appreciated.

[adamecr]

Hi @michaelglasson , I have an idea, just need to prepare and test the samples. Will get back to you

[michaelglasson]

Hi @adamecr, my colleague and I have confirmed some really good news about DMNEngine. In code, we found that we can pass arrays and other complex data structures to a literal expression. We can use a range of C# expressions and are not limited just to S-FEEL. This is great news. One thing we have not quite conquered is the possibility to use an array as a column input for a decision table. We tried a couple of things, but in each case, the exception we encountered was the unary test input == expression. Ideally, for example, we could put an expression into an input cell in a decision table and use the column input in the expression. For example, in Camunda 7, the default input variable is called 'cellInput'. We can use it in an expression in a table cell like, 'list contains("x", cellInput). We are working on it.

[adamecr]

Hi @michaelglasson ,
this is how to do it using the definition builder.

The "magic" is not to do the contains check in a rule cell, but to do it as a column definition (list.Contains(val)) and the rule just checks for true or false where needed.

You can also do and use the check in table output (for example when not a necessary part of decision table but might be used further in decisions tree) or in the expression decision.

Hope this helps, I will prepare the equivalent DMN if needed.

            var definition = new DmnDefinitionBuilder()
            .WithInput("list", out var inputList)
            .WithInput<int>("val", out var inputInt)
            .WithVariable<Boolean>("b", out var variableB)
            .WithVariable<string>("s", out var variableS)

            .WithTableDecision("Test", table =>
               table
                  .WithInput(inputInt, out var tblInputVal)
                  .WithInput("list.Contains(val)", out var tblInputContains)
                  .WithOutput(variableB, out var tblOutB)
                  .WithOutput(variableS, out var tblOutS)
                  .WithHitPolicy(HitPolicyEnum.Unique)
                  .WithRule("row1", r => r
                    .When(tblInputVal, "<5")
                    .Then(tblOutS, "\"less than five\"")
                    )
                  .WithRule("r2", r => r
                    .When(tblInputVal,"[5..10]")
                    .And(tblInputContains,"true")
                    .Then(tblOutS, "\"five to 10 + true\"")
                    .And(tblOutB, "list.Contains(val)")
                    )
                  .WithRule("r3", r => r
                    .When(tblInputVal, "[5..10]")
                    .And(tblInputContains, "false")
                    .Then(tblOutS, "\"five to 10 - false\"")
                    .And(tblOutB, "list.Contains(val)")
                    )
                 .Requires(inputInt)
                 
               , out var tblTest)
            .Build();

            var list = new List<int> { 2,4,6,8,12};
            var ctx = DmnExecutionContextFactory.CreateExecutionContext(definition);
            
            var result1 = ctx
                .Reset()
                .WithInputParameter("list",list)
                .WithInputParameter("val",1)
                .ExecuteDecision("Test");
            result1.FirstResultVariables[0].Value.Should().Be("less than five");


            var result6 = ctx
                .Reset()
                .WithInputParameter("list", list)
                .WithInputParameter("val", 6)
                .ExecuteDecision("Test");
            result6.FirstResultVariables[0].Value.Should().Be("five to 10 + true");
            result6.FirstResultVariables[1].Value.Should().Be(true);

            var result7 = ctx
                .Reset()
                .WithInputParameter("list", list)
                .WithInputParameter("val", 7)
                .ExecuteDecision("Test");
            result7.FirstResultVariables[0].Value.Should().Be("five to 10 - false");
            result7.FirstResultVariables[1].Value.Should().Be(false);

            var result12 = ctx
                .Reset()
                .WithInputParameter("list", list)
                .WithInputParameter("val", 12)
                .ExecuteDecision("Test");
            result12.HasResult.Should().Be(false);

[adamecr]

@michaelglasson - this is the DMN that can be parsed and executed in code.
I need to think a bit how to make it work with Simulator as you can't set the complex input value there
listForEngine.zip

[adamecr]

To make it work in Simulator, it's necessary to tweak it a bit :-(
Add the following classes to the Simulator project:

    public static class Extensions
    {
        public static bool Contains(this string s, int i)
        {
            return (s.Split(",").Select(i => int.Parse(i)).ToList()).Contains(i);
        }
    }

    public class CustomExecutionContext : DmnExecutionContext
    {
        public CustomExecutionContext(DmnDefinition definition, IReadOnlyDictionary<string, DmnExecutionVariable> variables, IReadOnlyDictionary<string, IDmnDecision> decisions, Action<DmnExecutionContextOptions> configure = null)
            : base(definition, variables, decisions, configure)
        {
        }

        protected override void ConfigureInterpreter(Interpreter interpreter)
        {
            base.ConfigureInterpreter(interpreter);
            interpreter.Reference(typeof(Extensions));
        }
    }

Then modify DmnService.ExecuteDecision method to use the CustomExecutionContext (just adjust the creation of the context to use the custom class instead of the default one:

 //create the execution context
 ctx = DmnExecutionContextFactory.CreateCustomExecutionContext<CustomExecutionContext>(definition, o => 
 {
    o.RecordSnapshots = true;
    o.ParsedExpressionCacheScope = ParsedExpressionCacheScopeEnum.Execution; //to ensure you can use object parameters with proxy type 
});

This would "inject" the string extension method Contains(int value) that will behave the same way as List<int>.Contains(int value).
So no need to change DMN, just the engine will here use the string param instead of List and the extension method will do the trick.

[michaelglasson]

@adamecr, I am not sure what I have done to cause you to devote so much of your 'hobby time' to my enquries, but thank you very much. I think I understand the approach you have outlined. I have downloaded the DMN and looked at the code you posted above. If we provide a list as an inputParameter, we can use it in the column definition expression. Very nice. I will certainly be able to use that. The simulator modification is not only helpful, but gives some insight into the code. So thanks for that. I have an additional comment which I will post separately so as not to confuse the issue.

[michaelglasson]

I also have a use case in which the list to be checked may be different for different rules.

As you can see, one rule test the input list to see if it contains anything in [A,B], while the second checks for anything in [C,D]. While the example is in FEEL, I am reasonably sure I can do the same thing in Dynamic Expresso.

I looked again at the DMN 1.3 spec and rediscovered the special '?' variable. Here is an excerpt that explains it.

One of the expressions in the inputEntry is a boolean expressions using the special ‘?’ variable and that
expression evaluates to true when the inputExpression value is assigned to ‘?’.

I see that Camunda engine works fine with either ? or the name of the input variable in a FEEL expression, but Iam not sure, but I wonder if DMNEngine recognises this symbol. I will check and get back to you.

[michaelglasson]

No, I don't think DMNEngine supports either the use of ? or the variable name in the inputParameter. Exception while parsing the expression list==list.Contains(5) and Exception while parsing the expression list==?.Contains(5). The issue appears to be that '==' operator is applied even when it is not needed. Perhaps what we should see is that when the inputParameter includes the '?' symbol or - perhaps better - the name of the input variable, then the cell is evaluated as a literal expression returning a Boolean.

[michaelglasson]

Now all I have to do is find the place in the DMNEngine code where this logic is handled.

[michaelglasson]

OK. The SfeelParser is the component that puts in left == right like this:

_// string in "" "", number,variable name, .. - compare for eq
conditionParts.Add($"{leftSide}=={part}");_

Maybe we can do something like:

• if there is a ? in the right side part, substitute the name of the input variable from the column heading and do not use the left side part at all OR
• if we see the name of the input variable from the column heading in the right side part, do not use the left side part at all.

The first one would stay true to DMN, but might stop us from using the c# ?: ternary operator. We could get around that by allowing us to select the expression language either for the whole column (S-FEEL or C#) or for each individual cell. If we select C#, then we need to distinguish between the ? in the ?: operator and the ? that stands for the input variable.

I can have a go at making the changes, but hopefully, you can spare the time to do the update? Let me know what you think.

[adamecr]

@michaelglasson - I'll think about it.
Personally, I'd like to avoid changes in SFeelParser as it can break a lot of things and the class itself was not designed as extendable.

I would say, that conceptually, you can achieve the change from left==right into right-> bool the way that the column expression will be true. This will transform to true==right and you can already use any variable (input) in the rule expression (right side) .

I'm out of computer, so can't give it a try, but I believe it's the way...

The midnight is approaching here, tomorrow I'll also show you how to extend the functionality for the right side.

these two things together should give you all you need, however, to manage the expectations, it will not be FEEL syntax

[adamecr]

Hi @michaelglasson ,
I gave it a shot and have to admit, we're reaching the limits of extensibility.

I used the custom execution context with following extensions:

protected override void ConfigureInterpreter(Interpreter interpreter)
{
    base.ConfigureInterpreter(interpreter);
    interpreter.Reference(typeof(Ext));

    Func<bool, bool> ev = (b) => b;
    interpreter.SetFunction("ev", ev);

    Func<string, List<int>> lst = (s) => s.Split(",").Select(i => int.Parse(i)).ToList();
    interpreter.SetFunction("lst", lst);
}

This helped me to play a bit with following decision table:

Actually, it's possible to have a DE (=Dynamic Expresso) expression in the rule, but there are limits due to SfeelParser logic. For example I was not able to do val>60, because it got parsed like true==val>60 and it's wrong. Using brackets (val>60) doesn't help because the parser thinks it's a range and fails. So I had to use hack like `!(val<60).

Bringing the function ev that just passes the bool expression helps, but there are still limits - for example any , that is not in string literal is understood as a "splitter" into sub-values (like in S-FEEL 1,4,5 meaning the value can be one of these).
That's the reason, why lst function takes and splits the sting to create List<int> instead of getting the params array.

But still, extensions only can do quite some work....

[adamecr]

... the next step means change of core. I will probably implement it in the next version, but for now, just publishing it here. The release will need bit more thinking & designing, testing and adding the test code, updating the doc etc., so more time needed.

I did tweek the SfeelParser.ParseInput method (only the HACK section is new):

  public static string ParseInput(string expr, string leftSide)
  {
      if (string.IsNullOrWhiteSpace(expr)) throw Logger.Error<DmnParserException>($"Missing expression");
      if (string.IsNullOrWhiteSpace(leftSide)) throw Logger.Error<DmnParserException>($"Missing left side of expression");

      if (Logger.IsTraceEnabled)
          Logger.Trace($"Parsing input expression {expr} for left side {leftSide}...");
      expr = expr.Trim();

      //#HACK issue #23 - bypass S-FEEL
      // use #: at the beginning of expression to bypass S-FEEL. In such case it's expected the the expression returns bool value
      //        use #? within the expression to place the left side ("column value") into the expression
      if (expr.StartsWith("#:"))
      {
          expr= expr.Substring(2);
          expr = expr.Replace("#?", leftSide);
          expr=expr.Trim();
          var cond = $"({expr})==true";

          if (Logger.IsTraceEnabled)
              Logger.Trace($"Parsed input expression {expr} for variable {leftSide} without S-FEEL: {cond}");
          return cond;
      }
      //#HACK END

      //custom functions translations
      foreach (var translation in CustomFunctionTranslations)
      {
          expr = expr.Replace(translation.Key, translation.Value);
      }
    .....

When the rule (cell) expression begins with #:, the parser don't do any S-FEEL and just use the expression as-is (=Dynamic Expresso syntax). It's expected the the expression returns bool value - eval of the rule.
The is only one manipulation of the expression - you can use #? within the expression to place the left side ("column value") into the expression.

I also used the custom execution context to provide additional functions (that's the valid extension point of engine):

public class CE : DmnExecutionContext
{
    public CE(DmnDefinition definition, IReadOnlyDictionary<string, DmnExecutionVariable> variables, IReadOnlyDictionary<string, IDmnDecision> decisions, Action<DmnExecutionContextOptions> configure = null)
        : base(definition, variables, decisions, configure)    {    }

    protected override void ConfigureInterpreter(Interpreter interpreter)
    {
        base.ConfigureInterpreter(interpreter);

        List<object> L(params object[] l) => l.ToList();
        interpreter.SetFunction("L", L);

        Func<System.Collections.IEnumerable, System.Collections.IEnumerable, bool> anyMatch = (arry1, arry2) => arry1.Cast<object>().Any(i => arry2.Cast<object>().Contains(i));
        interpreter.SetFunction("AnyMatch", anyMatch);
    }
}

The hack and these extension functions make this happen:

Hope this helps.

This is the test code:

       private static void TestCE()
       {
           var list = new List<int> { 2,4,6,8,12};
           var definition = DmnParser.Parse13ext(Path.Combine(Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location)!, "dmn", "listForEngineC.dmn"));
           var ctx = DmnExecutionContextFactory.CreateCustomExecutionContext<CE>(definition, c => c.RecordSnapshots = true);
           var result1 = ctx
           .Reset()
               .WithInputParameter("list", list)
               .WithInputParameter("val", 1)
               .ExecuteDecision("Test");
           result1.FirstResultVariables[0].Value.Should().Be("less than five");

           var result6 = ctx
           .Reset()
               .WithInputParameter("list", list)
               .WithInputParameter("val", 6)
               .ExecuteDecision("Test");
           result6.FirstResultVariables[0].Value.Should().Be("list contains val");

           var result10 = ctx
           .Reset()
               .WithInputParameter("list", list)
               .WithInputParameter("val", 10)
               .ExecuteDecision("Test");
           result10.FirstResultVariables[0].Value.Should().Be("rest");

           var result22 = ctx
           .Reset()
               .WithInputParameter("list", list)
               .WithInputParameter("val", 22)
               .ExecuteDecision("Test");
           result22.FirstResultVariables[0].Value.Should().Be("21-25");

           var result65 = ctx
           .Reset()
               .WithInputParameter("list", list)
               .WithInputParameter("val", 65)
               .ExecuteDecision("Test");
           result65.FirstResultVariables[0].Value.Should().Be("rest");

           var result80 = ctx
           .Reset()
               .WithInputParameter("list", list)
               .WithInputParameter("val", 80)
               .ExecuteDecision("Test");
           result80.FirstResultVariables[0].Value.Should().Be("71-90");

           var result101 = ctx
           .Reset()
               .WithInputParameter("list", list)
               .WithInputParameter("val", 101)
               .ExecuteDecision("Test");
           result101.FirstResultVariables[0].Value.Should().Be("rest");

           var result105 = ctx
               .Reset()
               .WithInputParameter("list", list)
               .WithInputParameter("val", 105)
               .ExecuteDecision("Test");
           result105.FirstResultVariables[0].Value.Should().Be("contains");

           //var list = new List<int> { 2,4,6,8,12};
           var result13 = ctx
              .Reset()
              .WithInputParameter("list", list)
              .WithInputParameter("val", 13)
              .ExecuteDecision("Test");
           result13.FirstResultVariables[0].Value.Should().Be("any");
       }
   }

[michaelglasson]

Those changes might just enrich DMNEngine enough to allow me to express all the rules in my use cases. I agree that hacks are not necessarily the best way to keep your code maintainable and that putting them in does not give as much satisfaction as finding a more elegant solution :).

I wonder if you have looked at DMN 1.3 Specification section 8.3.3 on page 77. It talks about 4 criteria by which an inputExpression may satisfy its corresponding inputEntry. Item d) says One of the expressions in the inputEntry is a boolean expressions using the special ‘?’ variable and that
expression evaluates to true when the inputExpression value is assigned to ‘?’.

I think that is the feature you are aiming for with the '#?' hack. If so, then, that may well be good enough for us at the moment.

[michaelglasson]

As an alternative to the SFeel vs Dynamic Expresso approach might be to enable the DMN feature that allows the expression language to be set for the whole DMN, for table column or for a table cell.

[michaelglasson]

And once again thanks for your engagement. I am an Australian enterprise architect looking for a solution for embedding rule processing within a larger .Net application. If we can do this, then we will be able to give our rule authors the graphic DMN rule creation experience, while allowing developers to embed those definitions into their code and evaluate them in place.

[adamecr]

I wonder if you have looked at DMN 1.3 Specification section 8.3.3 on page 77. It talks about 4 criteria by which an inputExpression may satisfy its corresponding inputEntry. Item d) says One of the expressions in the inputEntry is a boolean expressions using the special ‘?’ variable and that expression evaluates to true when the inputExpression value is assigned to ‘?’.

I think that is the feature you are aiming for with the '#?' hack. If so, then, that may well be good enough for us at the moment.

Yep, it's not exact 1:1 match I think but the idea is to provide you with similar functionality (to be able to use the "column value" in the "rule cell"

[adamecr]

As an alternative to the SFeel vs Dynamic Expresso approach might be to enable the DMN feature that allows the expression language to be set for the whole DMN, for table column or for a table cell.

I'm a bit tempted to think about implementing the FEEL support and support additional DMN elements like boxed expressions. But I'm afraid this is not a "month or two" task :-(

[adamecr]

And once again thanks for your engagement. I am an Australian enterprise architect looking for a solution for embedding rule processing within a larger .Net application. If we can do this, then we will be able to give our rule authors the graphic DMN rule creation experience, while allowing developers to embed those definitions into their code and evaluate them in place.

Cool! I'm always happy that something I do "for fun" can help the others to "do something real", so thanks for using it