Precompile bytecode

[Sebight]

Hello, I am optimizing our Berry workflow, and I am thinking of working with raw compiled Berry bytes that I will just send around and use.

As I understood, that's what .bec files are for. However, all I need is to get a buffer of the Berry bytes (for example, after calling be_loadstring) and later on load it.

Loading part

I suppose I could use the be_bytecode_load_from_fs function. However, I might not want to work with files in the future, so I could probably edit it to accept a byte stream.

"Saving" part

I basically need to get the bytecode, somewhere in the process of loading the code. I tried looking through the code, and I ended up somewhere around m_parser and be_parser_source; however, I'm still completely clueless. @s-hadinger Would you be able to give me a helping hand to locate a place where I can access the compiled bytecode buffer? If that's even possible though.

Thanks!

[s-hadinger]

Hi, the whole bytecode part was developed with file-system in mind for loading and saving. Replacing with a buffer would be an major change. That's why I implemented be_bytecode_load_from_fs so you can create a ad-hoc file-system, potentially mapped to a buffer. I did this in Tasmota, so you can download some bytecode from a URL into a buffer, and then load from this buffer as if it was a file in the file-system.

For Saving, you need to use compile() to generate the bproto and then call be_savecode(vm, fname); to save it to a file. Again there is no direct way to get a buffer, unless you create a pseudo-file that saves to a buffer.

[Sebight]

Is it a problem to keep the source code instead of bytecode?

That's what I'm trying to find out. It might be.

We send the raw Berry code (string) from our web interface into an ESP32 controller, which gets loaded using be_loadstring. These scripts could get quite large; therefore, sending the bytecode would save traffic and compilation times on the web.

I'm just snooping around, discovering all the possibilities of bringing the Berry size to as low as possible. Since we have the opportunity of working with Berry on the web (thanks to WASM), I thought this might be a good idea. I don't really know though.

[s-hadinger]

You could also consider minifying Berry code: remove all comments, and reduce all spaces/cr/tab to a single space.

[Sebight]

Hi, I already do that. I remove all comments and one-line the code (while keeping the spaces to make it a valid syntax).

By the way, I made a working demo with .bec in the browser yesterday.

At first, I hoped for a bigger improvement (I was delusional :D), but we are still looking at a 30% difference. This might be even increased by "obfuscating" all longer expressions (class names, functions, etc.) into smaller 3-letter symbols. However, this was only a demo, so I'm not sure if we will proceed with it or not. I might look into this again one day.

Thanks for help!

[s-hadinger]

Wow, impressive. I would think twice before obfuscating class names and functions. Berry relies on duck-typing, so class methods need to have their names preserved. It's completely safe for local variables, since they are mapped to registers and their names is discarded after compilation.

[Sebight]

Thanks for the info. I'll see what I can do until I break something. We need to bring this down as much as possible, but I don't want to be too crazy about it.

I'll write some follow-ups here when I get back to it. :)