Ignore some certain words?

[zhoujingfighting]

Hi, thanks for making this wonderful tool for parsing in Typescript!
I am writing a parser for spice syntax and face some issues which botherd me a lot

.param ffe=324 fdf=343 fdhhf=343 ththytfy=787 $ This is inline comment
dwdw=323213

I want to parse like this .param ffe=324 fdf=343 fdhhf=343 ththytfy=787 dwdw=323213
to ignore the comment and make the two lines continous

Below is what I wrote:

 const Param = createToken({
    name: "Param",
    pattern: /\.param|parameters/i
})
const StringLiteral = createToken({
    name: "StringLiteral", pattern: /[a-z0-9\_\-]+/i
});
const Equal = createToken({
    name: "Equal",
    pattern: /\=/
})
const NumberLiteral = createToken({
    name: "NumberLiteral", pattern: /[+-]?[0-9]+(\.*\d*)?(e-?(0|[1-9]\d*))?[a-zA-Z]*|[+-]?\.\d+(e-?(0|[1-9]\d*))?[a-zA-Z]*/i,
});
const Comment = createToken({
    name: "comment",
    pattern: /(\$|\;|\*).*/i,
    group: Lexer.SKIPPED
})

###########parser###########
 function expectSameLineSpaceSep() {
             return $.LA(0).startLine === $.LA(1).startLine || $.LA(1).image.startsWith('+') ||  $.LA(0).startLine < $.LA(1).startLine
}
$.RULE('ParseParam', (iag) => {
            $.CONSUME(Param)
            $.MANY({
                GATE: () => expectSameLineSpaceSep(), 
                DEF: () => $.SUBRULE($.generateParam)
            });
        })
$.RULE('generateParam', () => {
            $.CONSUME(StringLiteral);
            $.CONSUME(Equal);
            $.CONSUME(NumberLiteral);
        })

when I removed the comment, the parser worked well, but when I added comment as above, some thingwrong happend, anything I missed???

[bd82]

Hello @zhoujingfighting

I suggest you inspect the output of the lexer with and without the comment to ensure the comment token is properly.
This is the likely culprit for the different behavior.

[zhoujingfighting]

hi, @bd82 ,thanks for your reply
I tried your suggestion, it seems no distinction between with and without the comment
I found the error happend because the modelInstance definition conflicted with Stringliteral, the token definition and orders need to be adjusted

const ModelInstances = createToken({
    name: "ModelInstances",
    pattern: /[\r\n]\s*[a-zA-Z\_]\w*/i,
    line_breaks: true
});
const StringLiteral = createToken({
    name: "StringLiteral", pattern: /[a-z0-9\_\-]+/i
});
####token orders######
    const customizedTokens = [
        Param,
        Subckt,
      	End,
        ModelInstance,
        Model,
        NumberLiteral,
      	StringLiteral,
        LeftParentheses,
        RightParentheses,
        Equal,
        LineEnding,
	WhiteSpace,
        ValueExpression,
      	Comment
    ]

[bd82]

I see.

It seems a little strange to have two tokens which are distinguished by a preceding line break.
Are you certain that won't cause problems elsewhere?

[zhoujingfighting]

aha😂😂😂😂, maybe I need to test more strings, and I have not read fully the usage of many apis of chevrotain package, But I am going to use it in our project, hope to learn more about this package🎊🎊🎊

[zhoujingfighting]

hi, @bd82 , here araised a new doubt about special character;
for example : all the lines which started with '+' will be treated as a continous content of last line

.model name type par1=val1 
+par2=val2 par3=val3
.param
+par1=val1
+par2=val2 
+par2=val2 par3=val3
###### will be treated as below
.model name type par1=val1 par2=val2 par3=val3
.param par1=val1 par2=val2  par2=val2 par3=val3

I searched around discussions, but I am still confused about this, what do i need to do ?

• cumstomized token ?
• or anything else?

sorry to bother so much, please feel free to answer this🌷🌷🌷

[bd82]

whitespace sensitivity is problematic.
Have a look at this discussion
#1798

It may give you some ideas