Discussion: Insufficient Abstraction and Fragmented Ecosystem (Bring Your Own *)

[dotfede]

1 - Unnecessary, undesired duplication

So, everyone and their dog has their own version of:

• A PluginBase
• An EarlyBoundGenerator tool (some of them use the CrmSvcUtil, some don't)
• A lot of so-called helper methods that perform common tasks in a higher level of abstraction than the SDK provides.
• Many more things that I'm probably missing here

While in other platforms with different philosophies this might be considered "a rich ecosystem", IMHO, and I think most .NET devs will concur, this is unnecessary, useless duplication and should not exist.

Which leads me to the point of this discussion: The CDS SDK does not provide enough higher-level abstractions to write regular business logic on top of, and requires a LOT of things on top of it to allow regular "business logic developers" to just write business logic without getting overwhelmed by the technical implementation details of the underlying platform (which require architect-level knowledge to abstract away or cope with).

I have personally already resolved most of this, and my devs are able to just write business logic without even knowing the difference between targetting CDS or hitting a regular SQL database using a regular ORM, or even caring if their code is going to run in a CDS plugin or in an Azure Function or whatever. Some of them don't even know what an Azure Function is anyway.

That said, I've become a self-appointed CDS evangelist, due to how much I love the platform and the joy it brings me to be able to author and provide feature-rich business solutions based on it.

I would like everyone to enjoy the same level of productivity and ease of use of the platform my devs enjoy, and for reasons outside of my control I cannot open-source my own abstractions atm.

Some of the aspects where the CDS SDK abstractions are lacking:

2 - First of all, and I think most importantly: Don't make me remember things

There is a lot of places where a dev is forced to remember an implementation detail in order to perform a certain task. Examples include:

• What is 20? What is 40? I've seen these magic numbers spread everywhere people need to refer to plugin execution stage.
• Which EntityImages are supported by which plugins? does PostCreate support PreImage?
• What's the OrganizationResponse return type for this particular OrganizationRequest?
• What are the valid statuscodes for this particular statecode for this particular entity?
• How much is the maximum batch size for ExecuteMultipleRequest? Oh, great, my batch is bigger than that, now I have to split it into several smaller batches.
• This is a big one: What's the logicalname for this particular entity/attribute/relationship/optionset/etc? Which leads me to the next topic:

3 - The CDS development experience is too fond of string typing

while the dynamic-language-esque experience of late-bound might be okay for very specific use cases, the truth is that the compile-time-verified, intellisense-enabled, refactor-friendly, exploration-friendly early-bound is much more suitable to write complex, rich business logic on.

Given this fact, and given that the primary use case for this SDK is to write business logic, I think the entire API should be designed with an early-bound-first approach rather than late-bound.

This, of course, does NOT mean to change the existing API and break compatibility, but to provide additional abstractions on top of the existing API, and having the entire experience (from SDK to tooling to documentation) default to early-bound always, and providing access to the lower-level abstractions of IOrganizationService and the like for the exceptional use cases where this is needed.

By looking at IOrganizationService, and CdsServiceClient, we can quickly realize that these are designed with late bound in mind rather than early bound. Most APIs are lacking abstraction. Some examples:

• IOrganizationService.Create() returns Guid, when it should really return EntityReference, or rather, a typed EntityReference such as EntityReference<Account> or whatever entity happens to be created.

• All other CRUD methods from IOrganizationService either require string entityName or return a late bound entity.

• Associate() and Disassociate() require a Relationship object which requires a string relationship name, which is nowhere to be found except by opening make.powerapps.com and looking it up.

• The entire ColumnSet API is based on strings (rather than expressions or some form of strongly typed reference to entity properties)

• Plugins' EntityImage collections are really a dictionary with a string key, and there is no reference to the commonly used keys such as PreImage, PostImage and so on anywhere.

• Plugins' Message property is a string, and there is no reference to the commonly used values such as Create, Update and so on anywhere.

• The entire OptionSetValue and OptionSetValueCollection API is too low-level. It should be hidden behind enums in ALL cases.

There are probably many more examples of this, but I think I made my point clear.

4 - Noise is bad

All these deficiencies lead to a single logical result: business logic code that targets CDS is too "noisy", and is full of stuff that isn't really related to the business logic itself, but rather platform details and string based stuff that should not be there, which contribute nothing, and pollute the business logic code, and make it less maintainable, harder to write, harder to refactor, and most importantly harder to reason about.

Business logic code should strictly deal ONLY with the following aspects:

• Performing queries
• Manipulating entities and entity properties
• creating or modifying entities with their corresponding properties
• performing calculations or other business-related logic.

Business logic code should NOT have to deal with any of the following:

• Magic strings or magic numbers
• string typing of any sort
• Platform implementation details (OptionSetValue, logical names of anything (entities/attributes/optionsets/relationships/etc), OrganizationRequest, EntityImage / Message names)

This is what business logic code should look like. The API design of the SDK and the associated tooling, therefore, should have the goal of enabling developers to write such code.

[rappen]

Following this with interest, many valid points :) but you have added a bunch of links that just give me 404s, I'm assuming you're linking private repos?
I would wish you either opened them up so the links make sense or write here what you mean to provide with the links instead.
Thanks!

[dotfede]

@rappen Well I guess it doesn't hurt if I post my main abstraction here:

    public interface ICrmService: IFluentInterface
    {
        List<T> Fetch<T>(string fetch)                                                where T: EntityBase, new();
        IQueryable<T> Query<T>()                                                      where T: EntityBase, new();
                                                                                      
        T Retrieve<T>(Guid id)                                                        where T: EntityBase, new();
        T Retrieve<T>(Guid id, PropertySet<T> propertySet)                            where T: EntityBase, new();
                                                                                      
        T Retrieve<T>(EntityReference entityReference)                                where T: EntityBase, new();
        T Retrieve<T>(EntityReference entityReference, PropertySet<T> propertySet)    where T: EntityBase, new();

        T Retrieve<T>(EntityReference<T> entityReference)                             where T: EntityBase, new();
        T Retrieve<T>(EntityReference<T> entityReference, PropertySet<T> propertySet) where T: EntityBase, new();

        T CreateAndRetrieve<T>(T entity)                                              where T: EntityBase, new();
                                                                                      
        T Refresh<T>(T entity)                                                        where T: EntityBase, new();
                                                                                      
        EntityReference<T> Create<T>(T entity)                                        where T: EntityBase, new();
        void Update<T>(T entity)                                                      where T: EntityBase, new();

        void Delete<T>(T entity)                                                      where T: EntityBase, new();
        void Delete<T>(Guid id)                                                       where T: EntityBase, new();
        void Delete(EntityReference entityReference);
        void Delete<T>(EntityReference<T> entityReference)                            where T: EntityBase, new();

        void Associate<TLeft, TRight>(TLeft entity, Relationship relationship, TRight relatedEntity)            where TLeft: EntityBase, new() where TRight: EntityBase, new();
        void Associate<TLeft, TRight>(TLeft entity, Relationship relationship, List<TRight> relatedEntities)    where TLeft: EntityBase, new() where TRight: EntityBase, new();

        void Disassociate<TLeft, TRight>(TLeft entity, Relationship relationship, TRight relatedEntity)         where TLeft: EntityBase, new() where TRight: EntityBase, new();
        void Disassociate<TLeft, TRight>(TLeft entity, Relationship relationship, List<TRight> relatedEntities) where TLeft: EntityBase, new() where TRight: EntityBase, new();

        void SetState<TEntity, TStateCode, TStatusReason>(TEntity entity, TStateCode stateCode,
            TStatusReason statusReason)
            where TStateCode : struct, Enum
            where TStatusReason : struct, Enum
            where TEntity : EntityBase, IEntityWithStateCode<TStateCode>, IEntityWithStatusReason<TStatusReason>;

        void Assign<TEntity, TOwner>(TEntity entity, TOwner owner)
            where TEntity : EntityBase, new()
            where TOwner : EntityBase, IOwner;

        void Assign<T>(T entity, EntityReference owner) where T: EntityBase, new();

        byte[] RetrieveEntityImage<T>(T entity) where T : EntityBase, IEntityWithImage, new();
        byte[] RetrieveEntityImage(EntityReference entityReference);

        IBatchContext BeginBatch(int batchSize = 1000, bool multiThreaded = false);
        IBatchContext BeginTransaction();

        IFetchXmlRoot FetchXml { get; }
    }

and this is what a typical early bound entity generated by my Early Bound generator looks like.

Couple of things to notice here:

• All ICrmService methods accept and return early bound entities, late bound is disallowed by design by this abstraction.
• Access to late bound functionality is hidden behind an explicitly implemented interface (hidden from the view, and from Intellisense). If you really need to use late bound or IOrganizationService at some point, you have to do this:

IOrganizationService organizationService = (this.CrmService as IOrganizationServiceContainer).OrganizationService;
// use organizationService

• No string based stuff anywhere, no magic numbers. Just beautiful type-safe generic methods.
• Create<T>() returns EntityReference<T>, which I talk about here
• PropertySet<T> is a type-safe abstraction of ColumnSet, which works like this:

var contacto = crmService.Retrieve(id, new PropertySet<Contacto>(x => x.NombreDePila, x => x.Apellidos));

• SetState() uses constraints to guarantee that the enums you're passing to it actually belong to the entity you're trying to modify. You can't set the state of an Account and accidentally pass an enum value of Contact.OptionSets.StatusReason, for example.

• Lots of smaller quality-of-life improvements such as IOwner (which is implemented by SystemUser and Team, and IEntityWithImage, which tags entity with a primary image so you can call GetEntityImage() and so on.

• I still have to work on eliminating string typing for Associate(), and come up with a type-safe version of OrganizationServiceContext.Load(), which also uses strings.

Regarding the early bound generator, there's a lot to be said, it uses display names for class/property/enum names rather than logical/schema names, so you get a much more natural naming, which is also much closer to what you see in the frontend. This really helps reduce cognitive load and makes code look much nicer and cleaner in general.

Anyway, I could go on rambling about this forever, is there anything else you would like me to elaborate on?

[FixRM]

@BetimBeja I guess provided implementation is not optimal. I did an update to my implementation to take best from both worlds )

[BetimBeja]

@FixRM my work in progress:

[MattB-msft]

@BetimBeja would suggest you stick with the logical names for the fields vs their schema names as that may confuse the parser..
Though I am curious why you would not just generate a model for the target environment?

[BetimBeja]

@MattB-msft my actual implementation for ColumnSet is the following:

using Microsoft.Xrm.Sdk;
using Microsoft.Xrm.Sdk.Query;
using System.Collections;
using System.Collections.Generic;

namespace AlbanianXrm.EarlyBound.Extensions
{
    public class ColumnSet<T> : IEnumerable<string> where T : Entity
    {
        private readonly ColumnSet columnSet;

        internal ColumnSet(ColumnSet columnSet)
        {
            this.columnSet = columnSet;
        }

        public ColumnSet(params string[] attributes)
        {
            this.columnSet = new ColumnSet(attributes);
        }

        public ColumnSet(bool allColumns)
        {
            this.columnSet = new ColumnSet(allColumns);
        }

        public bool AllColumns
        {
            get
            {
                return this.columnSet.AllColumns;
            }
            set
            {
                this.columnSet.AllColumns = value;
            }
        }

        public void Add(string column)
        {
            columnSet.AddColumn(column);
        }

        public void AddColumn(string column)
        {
            columnSet.AddColumn(column);
        }

        public void AddColumns(params string[] columns)
        {
            columnSet.AddColumns(columns);
        }

        public IEnumerator<string> GetEnumerator()
        {
            return (IEnumerator<string>)columnSet.Columns;
        }

        IEnumerator IEnumerable.GetEnumerator()
        {
            return columnSet.Columns.GetEnumerator();
        }

        public static implicit operator ColumnSet(ColumnSet<T> typedDataCollection)
        {
            return typedDataCollection.columnSet;
        }
    }
}

And the existence of the attributes in the Early-Bound entities (always using logical name) is enforced using a Roslyn Analyzer. I have to reimplement most of the properties because I cannot extend the sealed classes so I am using encapsulation to give a similar API.

[dotfede]

@MattB-msft Quoting above:

I have to reimplement most of the properties because I cannot extend the sealed classes so I am using encapsulation to give a similar API

And this is EXACTLY the problem that sparked this entire discussion, and yet, almost two years later there has been ZERO progress about it and there is no indication that Microsoft even understood or acknowledged the problem.

Again, I'm really happy that I have abandoned the Power Platform never to return again.

[bthompson-365]

When using display names how will it / would it handle duplicate entity / attribute names as while logical names catch and disallows duplicates there is nothing stopping 2 entities and attributes from having the exact same display name. Also what determines the language that would be used.

Beyond that I don't really wish to comment - I don't use early binding as I find that while it may decrease development time when things go wrong with it in production they have a habit of going disastrously wrong. And given that we are no longer in a world of static managed releases that potential disaster could be at anytime.

[dotfede]

When using display names how will it / would it handle duplicate entity / attribute names as while logical names catch and disallows duplicates there is nothing stopping 2 entities and attributes from having the exact same display name.

let rec dissambiguate (index: int) (scope: Scope) (identifier: CodeGeneratorModel.Identifier) = 
   let newIdentifier = 
       match index with
       | 0 -> identifier
       | _ -> 
           identifier 
           |> Identifier.withValue (identifier.Value + index.ToString())

   let collision = 
       scope
       |> Seq.tryFind(collides newIdentifier)
   match collision with
   | Some e -> dissambiguate (index + 1) scope identifier
   | None -> 
       scope.Add newIdentifier
       newIdentifier

Besides, my tool allows to assign arbitrary identifier names to whatever entity/attribute/optionset/optionsetvalue/relationship/etc

Also what determines the language that would be used.

The default language of the user being used to download the metadata.
Also: metadata is cached to disk and stored in source control, and refreshed only when needed

Beyond that I don't really wish to comment - I don't use early binding as I find that while it may decrease development time when things go wrong with it in production they have a habit of going disastrously wrong. And given that we are no longer in a world of static managed releases that potential disaster could be at anytime.

Well, I've been running production code written this way for 5 years now, and there hasn't been a SINGLE case of whatever supposed "disaster" you're talking about.

On the other hand, at least 2 times in the past, I had to fix (rewrite to proper type-safe code) code from other vendors providing CDS-based solutions, who thought using a proper object model was "a bad idea" which of course didn't work and completely crashed and errored out the very same day they put it into production. So no, sorry, late bound is never an option if you need to write serious business logic.

Also: using a proper object model has nothing to do with reducing development time and everything to do with being able to understand what code is doing without having to guess or ask the person who wrote it in the first place, or, God forbid, read the "documentation". Statically typed code is refactor friendly, intellisense-friendly, lends itself much better to exploration, is easier to reason about, and in general creates MUCH less cognitive load when you have to read it.

Ask ANY .NET Developer in the world if using Dictionary<string, object> (which late bound basically is) is preferable to using a proper entity model with proper static types.

[bthompson-365]

You are writing code that is going to be used in environments 100% under your own control - that may be your world, it definitely isn't mine.

I take it by vendor you actually mean a different Microsoft / D365 partner - in which case if that got into production that is a testing / validation issue more than anything else.

And at that point I really will leave it - I find arguing with early binding zealots completely pointless but given that you were asking for comments, here is mine:-

Late binding should be treated equally as it does have its place and may be, depending on your actual requirements (say the ability to process data from multiple environments with different customisations), essential.

[dotfede]

You are writing code that is going to be used in environments 100% under your own control - that may be your world,

Yeah, that's what business applications are about. The home page for the Power Platform says "Business Application Platform", which means my use case is the primary use case for the entire platform.

That said, I don't see how not having control of the environment changes anything I've said here. Unless you don't have any entities whatsoever, in which case I can't imagine how you can provide any value at all.

it definitely isn't mine.

Your use case is 100% supported by current APIs. Late bound code is as good as it gets right now and there's not much that can be improved. In contrast, there's a lot of room for improvement for early bound, and I don't see how it hurts your use case if mine becomes better supported.

I take it by vendor you actually mean a different Microsoft / D365 partner - in which case if that got into production that is a testing / validation issue more than anything else.

Or the fact that 10+ people are needed to barely understand business logic written without proper abstractions and using lower level constructs such as string typing, dictionaries and hand-written fetch xml, instead of using a proper object model and LINQ.

Guess what? I replaced their entire team of 10 by just myself, proper architecture and my own tools (which by the way include a Roslyn-based late-bound to early-bound automatic translator)
And the end result was a working product, which their team of 10+ couldn't achieve. So I have very strong reasons to hold my point.

early binding zealots

Sorry, you're being disrespectful right from the start, by pointing out such an obvious, stupid issue as name duplication, I felt I was being treated like a clueless junior. Of course my tool deals with name duplication and with whatever you may want to come up and challenge me with. I've been using it for production code for 5 years and customers are pretty happy.

the ability to process data from multiple environments with different customisations

My point still stands that concrete business logic that covers a concrete business use case is better served by early bound. For more abstract / generalized uses late bound is okay. I do use late bound when I need it, but that's like only 5% against 95% of specific business oriented code.

Also: You've changed your argument from "early bound is dangerous because it crashes in production in unknown ways" (which, sorry, is complete BS) to "my use case is better suited by late bound", which is totally 100% valid and you're completely right that both options need to have the best possible quality of life and ease of use in the Power Platform.

[dotfede]

Platform Adoption

Another, super important issue that I want to raise here is platform adoption.

• How many .NET devs are there in the world?
• How many CDS devs are there in the world?
• Do we want to lower the entry barrier for all these .NET devs who have no idea what CDS is?
• Do we want to attract them into the platform and show them it's the best platform ever?

I recently had this experience of bringing a new dev to our team. He's pretty good at all things .NET, but he had never EVER worked with anything related to CDS or Dynamics before.

Because I've abstracted away most platform details behind regular .NET classes (model) and a simple interface (shown above), he was able to start writing business logic from the very first day. I didn't have to teach him what an OptionSetValue, because that's nowhere to be found except inside my abstractions. He also didn't have to look up what these magic strings are, or the logical names of anything, because again, these are nowhere to be found in any of my projects.

Also worth mentioning that regular .NET classes + 1 CRUD abstraction + LINQ is similar to what you find in the rest of the .NET Ecosystem (see Entity Framework, NHibernate, and the like)

On the other hand, code written against IOrganizationService and the like require a lot of CDS-specific knowledge, that most .NET devs won't have, and that I will go as far as to say that the lack of proper abstractions and the unfriendlyness of string typing will actually scare .NET devs away from the Power Platform, instead of attracting them.

[mathiasbl]

Every platform and framework comes with a learning curve. I can write .NET but that doesn't make me able to write an ASP.NET Core web app from day one. You need to lookup documentation and start learning. The same goes for CDS.

By abstracting everything behind your own classes or implementations. you are creating devs who know your abstraction layer but don't even know what is happening under the hood. The real interaction with the CDS platform.

Other than plugins you are able to interact with the platform through standards known inside the .NET world like OData and Web API.

[dotfede]

@mathiasbl can you show an example of simple business logic targetting CDS written using OData or Web API?

Last time I checked, Microsoft's examples for both involve a bunch of string concatenation and manipulating HttpRequest and manually generating JSON. Would you say that these lower-level constructs are suitable to write complex business logic on?

Also:

By abstracting everything behind your own classes or implementations. you are creating devs who know your abstraction layer but don't even know what is happening under the hood.

By this reasoning, ORMs are bad. Hell, even the entire .NET platform is wrong because you're creating devs who know how to write apps using C# but "don't know" what is happening under the hood with bytes moving around between RAM and CPU.

Should we all go back to assembly language?

[mathiasbl]

By this reasoning, ORMs are bad. Hell, even the entire .NET platform is wrong because you're creating devs who know how to write apps using C# but "don't know" what is happening under the hood with bytes moving around between RAM and CPU.

Should we all go back to assembly language?

I'm sorry but you're making this ridiculous. Every platform/framework has it's own API to interact with it. Do you make abstractions for every framework (EF, Azure SDK's,...) you use? No.

[dotfede]

I'm not sure what you mean but you can use HttpClient to interact with it?

So this entire SDK is pointless then? Why are we even wasting @MattB-msft's time here?

You can create your own classes that represent data being posted and queried.

Which leads me back to square 1. You actually need a proper MODEL before you can write any business logic at all.

Do you make abstractions for every framework (EF, Azure SDK's,...) you use? No.

Actually YES. I've written an abstraction that allows me to use the SAME entity model between CDS and NHibernate for an external SQL database, and MongoDB / Cosmos.

And, as a matter of fact, the whole point of this SDK is precisely to serve as an abstraction on top of CDS APIs. You're not arguing with me, but with Microsoft, who created the SDK to begin with.

You still haven't shown any piece of business logic code written against the Web API, so that we can compare that to what I've posted above.

[mathiasbl]

The SDK is there to be used if you'd want it. Nothing forces you to use it and it can only be consumed by .NET devs. What about devs who write python, ... they can do so by targeting the WebAPI.

Why do you need sample code? The business logic code won't be any different if you use the SDK vs just interacting with the WebAPI other than using different models / api to retrieve data

[dotfede]

What about devs who write python

I couldn't care less about them.

Nothing forces you to use it

Yes, my need to write clean, maintanable, refactor-friendly, easily navigable and discoverable, complex business logic code forces me to use the SDK because there's no other way to achieve that. Even so, I've had to add additional abstractions of top of it to make code much cleaner and concise.

The business logic code won't be any different

I fully disagree. I haven't seen ANY business logic code targetting CDS as clean and noise-free as I've posted here except the code that we write against these abstractions I've created. That's precisely what I'm trying to bring up in this discussion.

Why is everyone so eager to dismiss and reject my points instead of considering them? Does it bring so much cognitive dissonance?

[dotfede]

For the record: delegateas/XrmContext offers similar abstractions to mine, which further reinforces my useless duplication argument. That tool didn't exist (or at least it wasn't public) back in 2016 when I started writing my own. Then there's D365DeveloperExtensions which also overlaps a lot with my tooling.

[sergeytunnik]

Well that escalated pretty quickly.

Why is everyone so eager to dismiss and reject my points instead of considering them?

Is it okay that you make assumptions about everyone based on comments of actually three persons? This is rhetorical question.

[MattB-msft]

I am going to leave this thread open for a bit longer folks to collect more feedback..
I will say that we have internally had a number of discussions around the topics being brought here over the years and recently. It is quite challenging to provide the balance between the needs of the low code population, the General developer Population, and the Core ProDev folks.

thanks.

[mohsinonxrm]

Lots of good points by @fberasategui and @daryllabar . I think since @MattB-msft and his team are developing this new CDSServiceClient, it is good to bring up these points/issues/ideas and see how things can be improved.

Personally speaking, whenever I'm training a new person on Dynamics CRM SDK, I start with early bound entity. Why? because it is easy to understand, grasp and relate the concepts, the same thing that @fberasategui mentioned about abstracting things from .NET background devs.

Eventually, however, I move them to use the late-bound entity (which I personally use as well) as because most of the projects/repos are already implemented that way. Plus, on the client-side, javascript with OData WebAPI endpoint, you have to use schema names.

I think both have their place and should be maintained.

On the pre-stage/post-stage numbers, I think they can be improved, perhaps enums etc. but this is the kind of knowledge that you just have to learn once and it stays forever.

[MattB-msft]

Rather curious about the comments regarding Plugin Stages... to my knowledge, we removed public docs on those stage numbers years ago, and pretty much refer to them by their stage name's now...
Where are you seeing the need to use them? or are folks still finding them in Microsoft Public docs?

[dotfede]

@MattB-msft

public interface IPluginExecutionContext : IExecutionContext
    {
        //
        // Summary:
        //     Gets the stage in the execution pipeline that a synchronous plug-in is registered
        //     for.
        //
        // Returns:
        //     Type: Int32. Valid values are 10 (pre-validation), 20 (pre-operation), 40 (post-operation),
        //     and 50 (post-operation, deprecated).
        int Stage
        {
            get;
        }

        //
        // Summary:
        //     Gets the execution context from the parent pipeline operation.
        //
        // Returns:
        //     Type: Microsoft.Xrm.Sdk.IPluginExecutionContextThe execution context from the
        //     parent pipeline operation.
        IPluginExecutionContext ParentContext
        {
            get;
        }
    }

Since there's no corresponding enum or constants, you find code like this everywhere:

if (executionContext.Stage == 40)
    // ...

[MattB-msft]

Interesting, where did that come from? its that's not in our public doc's or samples over the last number of years. if it is, can you point me too it?

The pattern your inferring is from our v4 and v5 systems where we talked about managing step operation inside your plugin, then register for a given message w/out declaring a stage association and use switch statements to manage code execution on various stages. We actually documented it at that time and provided samples.

We modified that guidance in late v5 and v6 to move to explicit registration operations on stages and adjusted tools and samples accordingly.

With regard to not having a public enum for them, you are correct we do not have one. it is intentional in that we were discouraging the registration pattern that would require you to know them or understand how to apply them.

[dotfede]

Interesting, where did that come from? its that's not in our public doc's or samples over the last number of years. if it is, can you point me too it?

Simply Ctrl+Clicking in the IPluginExecutionContext symbol right into Visual Studio 2019 ;)

The pattern your inferring is from our v4 and v5 systems where we talked about managing step operation inside your plugin, then register for a given message w/out declaring a stage association and use switch statements to manage code execution on various stages. We actually documented it at that time and provided samples.

We modified that guidance in late v5 and v6 to move to explicit registration operations on stages and adjusted tools and samples accordingly.

Yeah... well sort of... but not quite.

The problem is, again (sorry to be annoying), lack of abstraction.

While the Plugin Registration Tool enforces registration with a specific Message/Stage/Entity, nothing stops you from registering more than 1 SdkMessageProcessingStep targetting the same IPlugin implementation (I.E for the same PluginType), and, whether bad practice or not, I've seen this very very frequently. I guess this is due to the fact that implementing IPlugin in a usable manner involves a lot of copypasta/repeated code/noise:

public void Execute(IServiceProvider serviceProvider)
{
    if (serviceProvider == null)
    {
        throw new ArgumentNullException("serviceProvider");
    }
    IPluginExecutionContext pluginExecutionContext = (IPluginExecutionContext)serviceProvider.GetService(typeof(IPluginExecutionContext));
    ITracingService tracingService = (ITracingService)serviceProvider.GetService(typeof(ITracingService));
    IOrganizationServiceFactory organizationServiceFactory = (IOrganizationServiceFactory)serviceProvider.GetService(typeof(IOrganizationServiceFactory));
    IOrganizationService organizationService = organizationServiceFactory.CreateOrganizationService(pluginExecutionContext.UserId);

Given this, the lazy approach is to if (context.Entity == "xxx" && context.Message == "Create" && context.Stage == 40) right below that, instead of creating another class and copypasting the same code over and over and over.

Also worth mentioning here: The PRT has several pain points that make it less than ideal for everyday, constant usage (I'd like to open a separate issue to discuss that if possible). These annoyances in the API and tooling make people take shortcuts which sometimes go against best practices, just to save some time and avoid unneeded clicks.

Of course, due to my obsession with "Everything must be strongly typed", I came up several abstractions, and my plugins ended up looking like this:

    [RunAsSystem]
    public class GenerarCuotasAdelanto : PostUpdateStep<GestionDeSocio>
    {
        protected override bool ShouldExecute() =>
            HasValueChanged(x => x.CantidadDeCuotasAAdelantar);

        protected override void Execute() => 
            new AdelantoDeCuotaBusinessProcess(CrmService).CrearCuotas(PostImage);
    }

Notice how I've encoded all plugin registration details in the type system:

• Entity Type: GestionDeSocio
• Stage and Message: PostUpdateStep<>
• Trigger Attributes: Plugin will fire only when attribute CantidadDeCuotasAAdelantar changes (HasValueChanged(expr) evaluates to expr(PreImage) != expr(PostImage))
• User Identity: [RunAsSystem]
• Which entity images are available for which combinations of Message and Stage, since each Step base class exposes only the available images.
  • PostUpdateStep<T> has Target, PreImage and PostImage properties (all of type T)
  • PreUpdateStep<T> only has Target and PreImage (also of type T, of course).
  • The same for all other base classes for commonly used messages and stages: PreCreate, PostCreate and so on.

All of this produces the following benefits:

• All common boilerplate, such as getting everything out from IServiceProvider is abstracted away.
• You don't need to remember anything. No magic numbers, no digging into dictionaries by string key to see if entity images are there. No guessing if this plugin has this image or that plugin has that image.
• All relevant objects, such as ICrmService, PreImage, PostImage, and so on are properties of this class, so you can have multiple methods that use those objects without having to pass them around as parameters. (Don't worry, instance lifetime is scoped to a single execution of IPlugin.Execute()).
• Due to the ease of writing such a class (basically Intellisense autocompletes every part of it), Plugin Steps end up really being "proxies" of business logic processes written elsewhere (in this example AdelantoDeCuotasBusinessProcess), which then is called by different plugins or even reused outside CDS in Web APIs or Azure Functions or whatever.
• Having all registration details in code enables my tooling to use reflection to obtain them, allowing single-click (literally) registration of plugin assemblies/types/steps right into Visual Studio.

The end result is that none of my devs have opened the PRT in years. Not even for debugging.

With regard to not having a public enum for them, you are correct we do not have one. it is intentional in that we were discouraging the registration pattern that would require you to know them or understand how to apply them.

Well, in the words of Bucky Fuller: "you can't change things by fighting the existing reality. If you want to change something, build a new model that renders the existing model obsolete". Since I got tired of telling my devs to follow best practices, I'm now using the type system to force them to. They cannot do anything outside the boundaries of my abstractions, because if they do, their code will not compile.

TL; DR: I guess the moral of the story is: both APIs and tooling need to be comfortable and ergonomic for people who want to follow best practices. If you want people to do the right thing, make it easy and obvious for them to do the right thing and hard and obscure for them to do the wrong thing, also known as "The Pit of Success".

[daryllabar]

@MattB-msft Can you explain more of what you meant by this statement?

With regard to not having a public enum for them, you are correct we do not have one. it is intentional in that we were discouraging the registration pattern that would require you to know them or understand how to apply them.

There are fundamental differences to how the plugin behaves/operates depending on how the plugin is registered. So what exactly are you trying to discourage and why?

[rappen]

@daryllabar I'm assuming discouraging from registering several plugin steps for the same plugin class. The plugins shouldn't have to check which stage they were triggered on, it shall be only one.
So you might create your PreValCreateContact and PreOpCreateContact plugins, both calling the CreateContact(MyStageEnum stage) method that has all the code for both stages.

[dotfede]

@MattB-msft @daryllabar @rappen so, do we all agree that SdkMessageProcessingStep should be 1:1 with PluginType? This is, every IPlugin implementation should only have 1 registered Plugin Step? I've been thinking about that for a while and so far I couldn't find an elegant design.

[bthompson-365]

We seemed to have been sidetracked by plugin registration and ancient history

It's worth looking at https://community.dynamics.com/crm/b/billoncrm/posts/crm-4-0-plug-in-registration-tool-walkthrough to see how registration of plugins worked in CRM 4 - (sorry the original blogpost with images has gone) but note that there is no mention of pre-validation, pre-operation and post-operation stages.

I remember when first playing around with Dynamic plugins (and we are going back 9 years or so) old code would have the context.stage logic in it as a plugin would be triggered multiple times within a request but you would only want the code to execute once. So the first step in any plugin was the defensive one of if context.stage isn't the one I want stop execution.

One question though - given that plugins are explicitly noted as not supported by this CDSServiceClient project why have we been sidetracked by it plugins.

As for @fberasategui last question - it's perfectly possible in a late binding world for a single plugin to have multiple steps attached to it if the plugin is used across multiple entities. Even with early binding it's possible that an plugin on an activity entity could be attached to multiple activity types provided only attributes within the activitypointer core were being accessed / modified. So I suspect the answer is that due to exceptions it's not possible.

[rappen]

@MattB-msft @daryllabar @rappen so, do we all agree that SdkMessageProcessingStep should be 1:1 with PluginType?

In most common cases, probably yes.
But I've also done a lot of generic configurable horizontal functionality, and in those cases it makes sense to register several steps for the same plugin class, and even the opposite of registering one step triggering on all entities (let's not go into performance conversation here).
And as you can imagine those plugins are [brace yourself] late bound... 😱

So I guess we're back to It depends.

[dotfede]

@bthompson-365

One question though - given that plugins are explicitly noted as not supported by this CDSServiceClient project why have we been sidetracked by it plugins.

I honestly don't think it's healthy at all to split the APIs and have a different way to write CDS plugins than other code targetting CDS running elsewhere.

If that is the case, I'll stay on Microsoft.CrmSdk.CoreAssemblies for a long, long time.

[jordimontana82]

The thing is because this package doesn't target .net standard (yet) and it is meant to be used for client applications only (as in the README), it is just not suitable to share production code between server side and client side yet (it's still in alpha so....). Specially if it is likely to introduce breaking changes in namespaces etc..

That doesn't mean we can't use it "to play with" , experiment, give feedback in pure .net core based applications. That's my take. For instance, FakeXrmEasy v2.0 will use this package, but both 1.x and 2.x will have to coexist at least until there is parity between the current full .net framework server side implementation and this new package (i.e. up until plugin infrastructure supports .net core).

Now, @MattB-msft and the team did an amazing job porting to .net core since I ran a regression test suite after porting it to 2.x and found very very little issues. :)

[magesoe]

This is an awesome discussion, and I have to say I agree with @fberasategui. Like you mentioned, Delegate tools offer some of the same abstractions that your tools seem to have with XrmContext and DAXIF, as an example this is a plugin registration in DAXIF.

RegisterPluginStep<Account>(EventOperation.Update, ExecutionStage.PostOperation, ExecuteAccountStuff)
    .SetDeployment(Deployment.ServerOnly)
    .SetExecutionMode(ExecutionMode.Asynchronous)
    .AddFilteredAttributes(
        x => x.Address1_City,
        x => x.Address1_Country)
    .AddImage(ImageType.PreImage, x => x.Address1_Country);

It seem you took it a step further, awesome!

I don't think Delegate's tools would have any merit for existing if CDS was early-bound first, which would be awesome - less stuff to maintain.

I can especially relate to your comments about .NET developers on CDS. With early-bound development and XrmMockup our raw .NET developers never needed to open a CDS environment to add business logic. The fields were created for them, they simply wrote code, added tests, and comitted the code. The build pipeline could take it from there. A lot of value without knowing the SDK.

That said, CDS is an old platform and has a ton of quirks. If all the quirks should be captured early-bound, then Microsoft would not add any features for a couple of years. The tooling maintainers of Delegate did once consider created a cds_Sales solution, where everything was build using custom entities, just so everything behaved as expected. I think @rappen once said he wanted Microsoft to "play by the same rules". If Microsoft did that, then we could have some amazing early-bound abstractions.

[pksorensen]

Abstractions are nice, just also remember that abstractions have to be easy to use. I have no experience with Dynamics before half a year ago when I started working with the Power Platform and CDS. However, I have 20+ years of experience with development, c#, .net, and been a user of .Net Core since we saw the first versions, so my expectations to these abstractions are that they follow what I am used to from that world.

The first thing that really jumps to my attention is that there is no Async/Await support. It's not a big issue, just surprising. https://twitter.com/pksorensen/status/1253393371051294723?s=20

Next, all the tooling required just to get started (xrmcontext,crmsvcutil). There is not of knowledge here that one has to overcome just to do something. In the past, I myself and so many other developers have just given up and talked with the rest API directly because we know how to do that and read the documentation about expected body/params.

I first recently discovered the value add of using the SDK (when this alpha was released and we could use it from .net core and azure functions) - because now I was using the same language that all of our CRM department was using and potential having resources that could help build new stuff.

So to make things easier for me and others I created an nuget package with a MSBUILD task that takes care of everything for me (using XrmContext as @magesoe mention).

So anyone using the new csproject format and working with integrations from outside CDS plugins, this is a super helpfull alternative to talking with REST API.

1. Add nuget reference:

    <PackageReference Include="DotNetDevOps.Extensions.PowerPlatform.MSBuild.CDSContext" Version="1.0.11" />

2. Add the following to csproject

   <GenerateCDSContext Condition="$(GenerateCDSContext) == ''">true</GenerateCDSContext>
    <CDSEnvironment>https://xxxxxxxx.crm4.dynamics.com</CDSEnvironment>
    <CDSAuthType>OAuth</CDSAuthType>
    <CDSEntitiesToExport>listmember,list,campaign,contact,account</CDSEntitiesToExport>
    <CDSSolutionsToExport>YourSolutionName</CDSSolutionsToExport>
    <CDSApplicationId>72b30e4b-ddf2-xxxx-xxxxxxxxxxxx</CDSApplicationId>
    <CDSTenantId>f4e9f427-973b-4f22-xxxx-xxxxxxxxxxxx</CDSTenantId>

On first build if a CDS folder do not exists, it will build your context and then on rebuilds it will update it.

Full Example:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netcoreapp3.1</TargetFramework>

    <IsPackable>false</IsPackable>

    <UserSecretsId>25868fd8-9f86-493b-a39b-b2b384b38e5f</UserSecretsId>
    <GenerateCDSContext Condition="$(GenerateCDSContext) == ''">true</GenerateCDSContext>
    <CDSEnvironment>https://danchurchaiddev.crm4.dynamics.com</CDSEnvironment>
    <CDSAuthType>OAuth</CDSAuthType>
    <CDSEntitiesToExport>listmember,list,campaign,contact,account</CDSEntitiesToExport>
    <CDSSolutionsToExport>DXRM01Integrations</CDSSolutionsToExport>
    <CDSApplicationId>72b30e4b-ddf2-42a6-94e4-8aa070bf12e1</CDSApplicationId>
    <CDSTenantId>f4e9f427-973b-4f22-9b36-05fadc8e2998</CDSTenantId>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.Dynamics.Sdk.Messages" Version="0.2.8-Alpha" />
    <PackageReference Include="Microsoft.Extensions.Configuration.UserSecrets" Version="3.1.5" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="16.6.1" />
    <PackageReference Include="Microsoft.PowerPlatform.Cds.Client.Dynamics" Version="0.2.8-Alpha" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.1.2" />
    <PackageReference Include="MSTest.TestFramework" Version="2.1.2" />
    <PackageReference Include="DotNetDevOps.Extensions.PowerPlatform.MSBuild.CDSContext" Version="1.0.11" />
  </ItemGroup>

</Project>

        [TestMethod]
        public void TestMarketingLists()
        {
            var sp = BuildServiceProvider();
            var client = sp.GetRequiredService<IOrganizationService>();
            using var cds = new CdsContext(client);

            var campaign = cds.CampaignSet.Where(k => k.TypeCode == Campaign_TypeCode.Telemarketingcampaign).ToArray();
            var contact = cds.ContactSet.FirstOrDefault();
            var account = cds.AccountSet.FirstOrDefault();

           
                client.Execute(new AddMemberListRequest()
                {
                    EntityId = contact.Id,
                    ListId = campaign.First().dca_telemarketinglist.Id
                });
            

Plugins

For plugins i also wanted something a bit more simple and lightweight for simple projects so i created a cli that can register them for me.

1. A typical project for me starts with using power platform CLI.

pac plugin init

This generate a sample project that has what I needed for plugins. (there is some knowledge to learn about the Plugin Context, but again I don't think I want developers to write plugins in CDS without knowing the basics, so I honestly don't feel that there is a big gap between what is provided and what is needed. Some extra extension methods for common stuff would of cause be nice).

2. Then I add the two steps above for getting my CdsContext in the project.

3. Adding a manifest.json file where I define the plugins that should be registered. The next step would be that the tooling generates this from attributes on my plugins in code.

{
    "assembly": {
    "plugin": {
      "id": "plugin",
      "path": "./bin/Debug/net462/PluginSample.dll"
    }
  },
  "plugins": {
    "GenerateAPIKey": {
      "entity": "apikey",
      "mode": "Synchronous",
      "stage": [
        "PostOperation",
        "PreOperation"
      ],
      "supporteddeployment": "ServerOnly",
      "invocationsource": "child",
      "type": "...fullplugin name...",
      "event": "create",
      "assemblyId": "plugin",
      "configuration": {
        "unsecure": "",
        "secure": ""
      }
    }
  }
}

4. I run my power platform extensions cli to register them (both at dev time, and from ci/cd)

ppe-cli.exe manifest --username <username> --solution <solutionname> --publisher <publisher> --environment-url <environment> --path <path 2 manifest.json"

A big thanks to @magesoe and the guys that made XrmContext, it really remove all the string usage, nice enums and strongly typed classes for the synced entities.

[FixRM]

There is spkl for that already. Almost an industry standard. https://github.com/scottdurow/SparkleXrm/wiki/spkl