bug: lexer codegen should validate token identifiers are valid

[maxcleme]

Hi,

I've been playing with Participle and I wanted to try to codegen feature, however the generated code seems to not escape some characters, leading to uncompilable source.

See the following example :

// ...
} else if match := match:(l.s, l.p); match[1] != 0 {
	sym = -47
	groups = match[:]
} else if match := match;(l.s, l.p); match[1] != 0 {
	sym = -48
	groups = match[:]
} else if match := match|(l.s, l.p); match[1] != 0 {
	sym = -49
	groups = match[:]
} else if match := match,(l.s, l.p); match[1] != 0 {
	sym = -50
	groups = match[:]
} else if match := match@(l.s, l.p); match[1] != 0 {
	sym = -51
	groups = match[:]
}
// ...

My rules used :

{
  // ...
  {":", `:`, nil},
  {";", `;`, nil},
  {"|", `\|`, nil},
  {",", `,`, nil},
  {"@", `@`, nil},
  // ...
}

As far as I can tell it comes from https://github.com/alecthomas/participle/blob/master/lexer/codegen.go#L251

While I do agree that I could rename my Rule name to make the whole thing works, I think it might be a good idea to come up with some strategy in order to avoid this when generating code.

I'm fine with writting a PR if you have a idea for this.

Cheers !

[alecthomas]

Ah nice catch, though they are not valid rule names as only identifiers can be matched by the parser. This is really a validation error, but a PR would be welcome to apply the validation.

[maxcleme]

@alecthomas

Alright about validation, but introducing the validation will create some breaking changes, isn't it ?

[alecthomas]

No, because using anything but identifiers has never worked. This has never come up before, I assume because nobody has tried to do what you did.