What is the preferred way to add a "builtin" model

[goto40]

I would like to provide a built-in model.

For example I am allowed to define types in my model (e.g. type mytype unisgned 32 bit).

I would like to provide a bullt-in model defining UINT32, UINt16 etc.

// model
val mydata1: builtin.UINT32
val mydata2: builtin.UINT16

with a builtin model defined as follows:

package builtin {
  type UINT16 unsigned 16 bit
  type UINT32 unsigned 32 bit
}

What is the preferred way to do this in langium?

• In xtext I could use LibraryConatainer to add a builtin model programmatically
• In textx I coud use, e.g. built-in models

Maybe you already have an example in your unittests I did not see...

[spoenemann]

In Langium 0.2 you can create a subclass of IndexManager and override initializeWorkspace to add an additional library document. In Langium 0.3 this functionality is moved to a new service WorkspaceManager (#377).

[goto40]

In Langium 0.3, in the WorkspaceManager: I wonder what is intended to be passed as URI to LangiumDocumentFactory.fromString when using an in-memory model (which has no corresponding file). Using a non-existing filename seems to work, but yields logs on the console in some situations:

The following happens when I move the cursor over a reference to the model built in memory (I used the URI URI.parse("built_in")).

[Error - 8:00:43 PM] Request textDocument/documentHighlight failed.
  Message: Request textDocument/documentHighlight failed with message: ENOENT: no such file or directory, open '/built_in'
  Code: -32603 

[spoenemann]

Good question, maybe we need to check the URI scheme before trying to load a document from file. Could you create an issue?

[goto40]

As sketched in #464 (thank you @msujew):

• I had to add the document to the cache by calling this.langiumDocuments.addDocument(doc);
• I also changed the URI to memory:///filename

Example for the overridden WorkspaceManager method loadAdditionalDocuments:

    protected loadAdditionalDocuments(_folders: WorkspaceFolder[], _collector: (document: LangiumDocument) => void): Promise<void> {
        const model_text = `
            package built_in {
                rawtype uint32 UINT 32
                rawtype uint64 UINT 64
            }        
        `
        const doc : LangiumDocument = this.factory.fromString(model_text, URI.parse("memory:///built_in.item"));
        _collector(doc);
        
        // add doc to the cache, as suggested in https://github.com/langium/langium/issues/464
        this.langiumDocuments.addDocument(doc);
        return Promise.resolve();
    }

[goto40]

I further tried to unit-test my built-in model. For that I first made a test w/o the buit-in model, which works as expected (I adapted my "load model" from langium/lib/test/parseDocument

const services = createItemLanguageServices().ItemLanguage;
// ...

    const doc = services.shared.workspace.LangiumDocumentFactory.fromString(
        `
            package first (property_set first.props) {
                property_set props {
                    property optional description: STRING
                }
                rawtype my_int UINT 32
                struct First {
                    scalar x: my_int (.description="x value")
                    scalar y: my_int (.description="y value")
                }
            }        
        `,
        URI.parse("memory:///built_in.item")
    );

    await services.shared.workspace.DocumentBuilder.build([doc]);
    const model = doc.parseResult.value as Model; 

    // here, I test the content...
    // ...
    expect((struct.attributes[0] as ScalarAttribute).type.ref?.name).toEqual("my_int") // OK!

However, when adapting the test to use my built-in model (I added services.shared.workspace.WorkspaceManager.initializeWorkspace([]);), I get unresolved references (I checked that my method loadAdditionalDocuments is called as expected; see above) ...

    services.shared.workspace.WorkspaceManager.initializeWorkspace([]);
    const doc = services.shared.workspace.LangiumDocumentFactory.fromString(
        `
            package first (property_set built_in.props) {
                struct First {
                    scalar x: built_in.float64 (.description="x value")
                    scalar y: built_in.float64 (.description="y value")
                }
            }
        `,
        URI.parse("memory:///built_in.item")
    );

    await services.shared.workspace.DocumentBuilder.build([doc]);
    const model = doc.parseResult.value as Model; 

    // here, I test the content...
    // ...
    expect((struct.attributes[0] as ScalarAttribute).type.ref?.name).toEqual("float64") // fails!

Probably I just made some mistake on calling the API - I surely did not understand the details of how the DocumentBuilder and the reference resolution work together... I would greatly appreciate some hint, how to proceed (maybe a link to some example I missed or some other source of information). Thank you for your time.

[msujew]

Hey, @goto40,

I'm wondering about the URI you are using for your doc document. Is memory://built_in.item the same URI you are using for your built-in library document, which is loaded during the WorkspaceManager#initializeWorkspace([]) call? If it is, it would override the already existing library and insert itself at this place, making it impossible to resolve cross references to the old file. Is this the case?

Anyway, you made me aware of all the trouble it currently takes to unit test your language. Although we have a few utilities (most notable parseHelper exported from langium/test), we should try to improve on that for Langium 0.4.

[goto40]

@msujew, thank you for the quick reply.

You are absolutely right - I accidentally reused the same name for my test model as for my builtin model (copy-paste-error). I fixed this. However, this did not solve my problem...

I rewrote the test with parseDocument from langium/test.

I noticed, that I have to call services.shared.workspace.WorkspaceManager.initializeWorkspace([]); in order to call my workspace manager (and to load the builtin model). This is unclear to me (why it is not called automatically).

After initializing my workspace, I saw that my index (AstNodeDescriptionProvider) and my ScopeComputation get called correctly (with my test model file and with my builtin model - checked via console.log)?

However, when calling my ScopeProvider manually I do not get any elements from the builtin model... I will further investigate.

    const scope = services.references.ScopeProvider.getScope((struct.attributes[0] as ScalarAttribute),"ScalarAttribute:type");
    scope.getAllElements().forEach( (value, index) => {
        console.log(`${index}: ${value.name}`)
    });

[goto40]

Note: in case of multiple languages it is important to use a valid URI (else the extension is not extracted correctly; langium 0.4.0). Thus, use memory:///filename.ext in the example above.