Performance of large dictionaries

[LordGolias]

Hi,

When compiling something like this

program = _{ SOI ~ (expr ~ ";")* ~ (expr ~ ";"?)? ~ EOI }
expr    =  { prefix* ~ primary ~ (infix ~ prefix* ~ primary)* }
infix   = _{ binary_op }
prefix  = _{ unary_op }
primary = _{ value }

unary_operator = {
    "!"
  | "+"
  | "-"
 ...
}

with unary_op containing ~1k cases, it takes a prohibitive amount of time to compile.

Does anyone know if this is by design?

[tomtau]

Partially, I saw this older issue: #169 (comment) you can possibly improve the compile time by cargo-expand and using that generated parser code directly instead of generating it on-the-fly for every compilation.
Anyway, feel free to open an issue with more details if the compile time performance is very prohibitive.

[LordGolias]

Expanding mitigates this, although compile times is still pretty damming. I believe the issue is that pest tries to compile the whole thing, which makes the binary quite large. In particular, the generated code for a unary op

#[inline]
#[allow(non_snake_case, unused_variables)]
pub fn unary_operator(
    state: ::std::boxed::Box<::pest::ParserState<'_, Rule>>,
) -> ::pest::ParseResult<::std::boxed::Box<::pest::ParserState<'_, Rule>>>
{
    state.rule(Rule::unary_operator, |state| {
        state
            .match_string("!")
            .or_else(|state| state.match_string("+"))
            .or_else(|state| state.match_string("-"))
            ... 1.5k times ...
            
    })
}

which is likely causing a long compilation. I wonder if these expressions could written with a static hashset and an "contains", which is much smaller and easy to compile (and similarly performant).

To give an idea of the scale, the vocabulary is composed by 1.5k unaries, 1.5k binary ops and 300 nullaries. Most of them are not part of the "core" language, but like the standard lib. For this reason, does it make sense to include them on the .pest?

I.e. apart from a small set of symbols, everything is read left to right. Could it make sense delegate the creation of expressions (binary or unary) to runtime, thereby avoiding compiling all these symbols into the binary?

[tomtau]

which is likely causing a long compilation.

yes, it seems to be a likely cause. There's also this experimental project: https://github.com/Mubelotix/faster-pest that, besides a faster runtime, may also compile faster.

To give an idea of the scale, the vocabulary is composed by 1.5k unaries, 1.5k binary ops and 300 nullaries. Most of them are not part of the "core" language, but like the standard lib. For this reason, does it make sense to include them on the .pest?

I guess there are pros/cons of each approach?

I.e. apart from a small set of symbols, everything is read left to right. Could it make sense delegate the creation of expressions (binary or unary) to runtime, thereby avoiding compiling all these symbols into the binary?

Perhaps; if they are some precedence relationships among them, it could perhaps be combined with Pratt parsing.