Use VSG as a parser

[jfcliche]

I am looking for a Python-based VHDL parser that mostly is up to date with VHDL2008. Many existing parsers I have tried fail with the codebase in my project, but VSG seems to be doing fine with it. I just need to extract entities, ports and associated comment blocks and comment lines to automatically generate documentation for my Sphinx VHDL Domain plugin.

Is the VSG parser accessible in python, what level of parsing can it do, and is there API documentation for it?

So far I have been able to tokenize a file with vhdlFile.vhdlFile(lines).get_all_tokens().get_tokens() by looking at the source code and some guesswork, but is there some method/class that I could access to get more info on the code structure while still having access to the comments?

I suspect that VSG does not need to deeply understand the code to lint it and might not need do do much more processing passed the tokenizing, but my guess is there is a bit more than what I have seen by a quick look around.

Thanks for the great tool!

[jeremiah-c-leary]

Morning @jfcliche ,

Is the VSG parser accessible in python

It looks like you found vhdlFile which contains the lexer and parser. vhdlFile calls the lexer, classifies whitespace and comments and then calls the parser. At the moment this would be the best entry point into the parser.

, what level of parsing can it do

The parser was written to follow the LRM and is contained in the vsg/vhdlFile/classify directory. Each file in the classify directory is a production in the VHDL LRM. Most productions are covered. The expression and condition productions were a challenge because VSG operates on a single file and does not necessarily have all the information it needs to completely parse a file.

and is there API documentation for it?

Unfortunately at the moment there is not an API documented for it. Maybe this is an opportunity to create one though.

So far I have been able to tokenize a file with vhdlFile.vhdlFile(lines).get_all_tokens().get_tokens() by looking at the source code and some guesswork, but is there some method/class that I could access to get more info on the code structure while still having access to the comments?

VSG keeps all tokens in a list which represents the entire file. Unfortunately this does not show any kind of structure of the file. This has been a challenge sometimes when trying to write certain rules, but I have developed a methodology to move through the structure. Depending on what you want to do it should be fairly easy to get the information you want.

I just need to extract entities, ports and associated comment blocks and comment lines to automatically generate documentation for my Sphinx VHDL Domain plugin.

That sounds intriguing. Is this something you are working open source?

There are methods on vhdlFile that will help in extracting information from the token list. For example, if you wanted to get all the entity identifiers you could use this method:

vhdlFile.get_tokens_matching([vsg.token.entity_declaration.identifier])

This would return a list of Tokens of Interest (ToI) objects. You can get the value of the identifier by using .get_tokens[0].get_value().

The vsg_parser tool in the bin directory will show you how each line is parsed.

For example with this code:

  1
  2 entity FIFO is
  3
  4 end entity;

vsg_parser will generate the following:

--------------------------------------------------------------------------------
0 |
--------------------------------------------------------------------------------
1 |
<class 'vsg.parser.blank_line'>
--------------------------------------------------------------------------------
2 | entity FIFO is
<class 'vsg.token.entity_declaration.entity_keyword'>
<class 'vsg.token.entity_declaration.identifier'>
<class 'vsg.token.entity_declaration.is_keyword'>
--------------------------------------------------------------------------------
3 |
<class 'vsg.parser.blank_line'>
--------------------------------------------------------------------------------
4 | end entity;
<class 'vsg.token.entity_declaration.end_keyword'>
<class 'vsg.token.entity_declaration.end_entity_keyword'>
<class 'vsg.token.entity_declaration.semicolon'>

There are methods for extracting ranges of tokens based on a left and right token:

vhdlFile.get_tokens_bounded_by(vsg.token.entity_declaration.entity_keyword, vsg.token.entity_declaration.semicolon)

and some methods at a higher level of abstraction:

vhdlFile.get_subprogram_body()
vhdlFile.get_function_subprogram_body()
vhdlFile.get_procedure_subprogram_body()

I'm sure we can extract the information you are looking for. vhdlFile may already have the methods you need, and if it does not then I can add some.

Let me know how you would like to proceed.

Regards,

--Jeremy

[jfcliche]

ok thanks. Maybe that file should be referred to in "Setting up development Environment" on readthedocs?

For automatic versioning, colleagues have suggested versioneer (https://pypi.org/project/versioneer/) and it has worked for me for github/bitbucket hosted package repos, but I have never subbitted packages to pypi.

[jeremiah-c-leary]

I had wondered if I should include that file.

Thanks for the tip on versioneer. I will take a look at it.

[jfcliche]

Today I learned how to create packages with pyproject.toml instead of setup.py and to build and publish Sphinx docs using github actions. Here is my parsed-vhdl-domain doc page: https://jfcliche.github.io/parsed-vhdl-domain/ which includes an example of parsed VHDL.

BTW if you have an idea for a better name for the package I'm open to suggestions. I don't particularly like the name I have chosen. vhdlDomain and sphinx-vhdl already exist. Would sphinx-vhdl-domain, or VHDLDom, or vhdl-sphinx-domain aka VSD, friend of VSG?
JF

[jfcliche]

Hi. I have been looking around on automatic versionning and it seems like versioneer is no longer maintained since that capability is now handled by setuptools if you use a pyproject.toml-based install and use the setuptools-scm dependency. Just takes a couple of lines in the .toml file. I pushed a new version of parsed-vhdl-domain that uses it. I learned something useful here.
JF

[jeremiah-c-leary]

Evening @jfcliche ,

@alonbl is graciously helping to upgrade the tooling for VSG as part of PR #1141. He is including tox and pre-commit. It is some interesting stuff you might be able to use.

BTW if you have an idea for a better name for the package I'm open to suggestions. I don't particularly like the name I have chosen. vhdlDomain and sphinx-vhdl already exist. Would sphinx-vhdl-domain, or VHDLDom, or vhdl-sphinx-domain aka VSD, friend of VSG?

I am horrible at naming things. VSG is pretty good, but ELFWS is horrible. Are there any conventions for naming sphinx domains?

--Jeremy

[jeremiah-c-leary]

Morning @jfcliche ,

I found the issue with the failing tests.

The attribute .iId is missing on the last parameter:

577             replace_token(lOpenParens[-1], lTokens, token.aggregate.open_parenthesis(), iId=lTokens[lOpenParens[-1]])

it should be:

577             replace_token(lOpenParens[-1], lTokens, token.aggregate.open_parenthesis(), iId=lTokens[lOpenParens[-1]].iId)

[jfcliche]

Thanks. My bad. Fixed and pushed on my fork,

[jfcliche]

Hi. I have submitted PR #1309 that enables VSG to be used as a VHDL parser.
Cheers,
JF