Deep state

[mikaelho]

Hello Ken

Would you be interested in making the state deeply reactive by default? If yes, we could look into integrating my syncx project, with features like:

• Deep changes detection
• Support for dataclasses, pydantic (although probably needs some updating for v2 of pydantic)
• with context manager to apply or roll back multiple changes
• History, undo/redo

All the code and dependencies are pure python, but would need to experiment to see if MicroPython is feasible for a limited feature set (e.g. no dataclasses).

[Neon22]

Not my project but could you explain in more detail how it would help and where it would replace/enhance the current reactivity ?

[mikaelho]

Thanks, a fair question.

Based on the documentation, currently the UI is only redrawn automatically if changing a value on the state dict directly. Changing a value of a nested dict, list or similar is not detected, and will not result in the UI update. For this, using a context manager is required, with explicitly stating the changed values. This is workable, but extra effort for the developer, and arguably not as clean as the rest of the reactive state usage.

In short, from the point of view of the developer, there is an artificial difference between modifying the root items and the nested items of the state, which need to be done differently even if the developer intent is the same in both cases: modifying the state.

With the proposed approach, we would automatically detect the changes also to nested elements, with the path to the changed item also provided automatically.

As additional benefits, we could have transactional all-or-nothing behavior for multiple changes, and an option for letting the user undo any changes to the state (and thus to the UI).

While the benefits seem clear to me, personally, I could very well understand if it is not worth the additional code/dependency for the project, at least for the moment.

[Neon22]

Ahh. IIRC Vue2 had a similar issue. I do a lot of UI with arrays of data represented as SVG elements and Vue2 did not notice any changes to the contents of an array - which was dynamic based on further UI changes.

In the end I abandoned Vue2 because of the manual overhead for my case. Vue3 fixed this with deeper introspection? and a complete overhaul as well.

I haven't experimented yet with puepy to see how it work in my use case.. soon...

Does this sound like an example of what you mean ?

[mikaelho]

Yes, sounds like the same thing. 👍

[kkinder]

Vue2 did indeed have the same issue. As I recall, watchers had the same issue though you could specify deep watchers in Vue2.

[kkinder]

... actually the fact that it was a limitation in Vue2 sort of validated, in my mind, it being a reasonable limit.

[Neon22]

well I accept that :) but it did mean I halted my work in Vue2. I'd very much like a way to watch an array of values and update a related svg section if it was possible. I don't mind what the mechanism is though :)

At the moment I'm doing it all in ltk with callbacks and global variables. deterministic for sure and no mysterious 'magic' happening in the BG :)

[kkinder]

What an intriguing project. The lack of dataclass support in micropython does indeed rule it out for micropython support. I wonder if I could add it as an optional dependency? Something like a plugin you can install if needed for larger projects?

That could get messy though if you're commingling code that expects deep observation with code that does not, so it would need to be explicit.

I'll tinker around with it. I definitely want a deep observer feature of state changes but it seemed too complex for the effort and I gave up. Having a plugin you can install to get that functionality for Pyodide might be ideal.

[mikaelho]

For clarity, syncx does not require dataclass support, it just knows what to do if you give it dataclasses, and availability of dataclasses can be easily checked at import time.

The main point would be enabling change tracking in nested dicts, lists and sets, as well as in any __dict__-driven custom class.

As it seems that this might be interesting for the project, for my own education I will start with checking the feasibility of using the core dependencies (ProxyTypes and dictdiff) on MicroPython.

[mikaelho]

Does not look like an easy road.

Of the two dependencies, trying to just run a "Hello, World" in MicroPython with the dependencies available as wheels:

• dictdiffer starts, but trying to use the package fails because of missing collections.abc.
• ProxyTypes fails to start, with an unhelpful JS error: Uncaught (in promise) <generic error, no stack> custom.js:168

With enough time, it is probably feasible to vendor the packages in and make the necessary changes. dictdiffer project could possibly even take in any reasonable changes, ProxyTypes would probably be better kept as vendored.

[Neon22]

These would be good to raise on the discord to ntoll, jellglass. We might get some traction as ntoll has connections to micropython chief dev.

[mikaelho]

These would be good to raise on the discord to ntoll, jellglass. We might get some traction as ntoll has connections to micropython chief dev.

Hmm. My first instinct would be to work on the dependencies, neither of which is overly complex, rather than try to "expand MicroPython to be more like CPython". But maybe you have more insight into MicroPython project goals and what kind of changes they are willing to pursue?

[Neon22]

Damien has been most helpful in adding some things, he thinks are correct, to the wasm port of micropython.
In common with other ports there are specific allowances for each architecture. He is treating the wasm port as its own special platform. E.g. the stack got enlarged a lot recently which would be bad on most microcontrollers.

I think its a safe request. You just never know.

[kkinder]

Just following up on this, I think the best approach here is to have an optional dependency for this on the Pyodide runtime. It'll be a little tricky because I don't want to expand the runtime/downloaded codebase too much for optional dependencies, so the idea would be to put a really minimal hook into PuePy, then if syncx and maybe a puepy plugin are installed (via micropip), have the option to use them the same way you can optionally use a router.

So then, my concern then is, I don't have two behaviors for component developers. It would be weird if state works one way for when syncx isn't installed, then another if it is, but as a component developer you have no idea which it is.

One solution there is to have a per-component flag: something like use_deep_state = True. But that seems like yet more complexity, which I don't love, and it possibly introduces the same problem if you ever do component inheritance.

[mikaelho]

As soon as I have some time, I could take a look at which parts of the dependencies of syncx are really needed, and try to see if I could build a bespoke light-weight no-dependencies mpy-compatible version for PuePy.

[mikaelho]

Thanks! I did experiment with __dict__, but must have gone wrong at some point.

It should not be a big deal to version ProxyTypes for this purpose, following the pattern you shared in your experiment.

[mikaelho]

Just sticking to basic data types is possible as a fallback position, but in my experience you very quickly want to use objects Decimal, datetime etc. as data, so it would be convenient to enable them as well.

[mikaelho]

I wondered why the trick with __dict__ worked, and seems it is simply because MicroPython does not call getattr for accessing any of the "known" dunder attributes like __class__, __dict__ etc.

The downside of this is that any wrapped object will not look like the wrapped object, which is half of the point of the original proxy types. 🤔

[Neon22]

I'm pretty sure getattr is NOT used by pyodide (or any python interpreter using the peps) for the reason stated above. I.e. getattribute is used when getattr fails and getattr is used when direct access fails. I.e. a regular iv of a class. But I am not a language lawyer...

[mikaelho]

In any case, this should be secondary for our purposes here, as deep state is more interested in detecting changes, so I will continue with that.

[mikaelho]

Ok, here you can see a running example of where I am with this, on MicroPython.

The main.py runs a set of test cases, reactive_data.py is the file with all the "real" code, and test_case.py is just a quick helper to run tests in the browser (mainly it checks that the change callback is called with the expected value).

Where we are:

• Basic dicts, lists and sets are more or less supported.
• You start tracking a state object like this: tracked_state = track(state, callback).
• Then you get a callback with path to the change, whenever the state is changed at any level.
• The tracked state object can also be used as a context manager, in which case the callback is called with all the changes at the end of the context block.
• There is also a prune function that can be given all the changes, and it will return paths to changes that are "closest to root".
• I did not bring in dictdiff at all, as the change diffs are probably not that central for PuePy.

Some MicroPython fun I had along the way, in addition to the previous topics discussed here.

• Functions do not have __name__s
• lists do not have __setitem__
• list.append works in reverse order?

[kkinder]

I did an experiment with classes. I wasn't quite sure I understood the prune thing, so I just did this for the moment:

class Model1:
    def __init__(self):
        self.model2 = Model2()


class Model2:
    def __init__(self):
        self.some_attr = 1


model = Model1()
def cb(*args, **kwargs):
    print("callback", args, kwargs)
tracked = track(model, cb)
model.model2.some_attr = 2

A single callback was triggered in Pyodide (not quite the parameters I was expecting, but that's ok). But only one. In MicroPython, I got a traceback:

PythonError: Traceback (most recent call last):
  File "<stdin>", line 19, in <module>
  File "reactive_data.py", line 232, in func
AttributeError: 'Model2' object has no attribute '__setattr__'

I think it's probably fine if we just can't track classes or certain types, but it would have to be handled gracefully.

Either way, quite promising though.

[mikaelho]

Regarding the introspection, that is an issue with MicroPython that I mentioned earlier, and have not been able to circumvent (yet?).

Custom classes had an issue, which I left for next, just to get a first version out.

In your experiment, I would expect only one callback, with path info showing that the change happened "within" model1.

[kkinder]

You're right, it should only be called once. I wasn't thinking about it. Nevermind.

What are your plans for this module? Further refinement? Contribution to PuePy? If you want to contribute it, I think it would be a valuable addition, but I'm not sure how to go about it. I think there are three possible options:

1. Make deep reactivity standard across all PuePy projects/components
2. Make deep reactivity optional environment-wide
3. Make deep reactivity optional per-class

I'm inclined to rule out number two just because it could lead to some debugging headaches, if for example you use someone's code and their environment setting isn't the same as yours (as might be the case in a component library).

Per-class is safer, since you're the one writing the class, but it adds complexity to the codebase to have to think in two modes and have two reactivity models.

Making it the project-wide default, for everything, might be the way to go. The only real downside is that, especially for state that includes any kind of complex object, the compromises made for MicroPython might cause unexpected/unwanted behavior.

Hmmmm.... 🙇‍♂️

[mikaelho]

This version was created with the idea that it could be contributed to PuePy. While a MicroPython version might have some general use outside PuePy, I would now like to focus on just whatever is needed to make it valuable to PuePy.

I fully agree that consistent operation across platforms would be desirable, as that would, for its part, enable quick iteration on local CPython, before deploying and finalising on browser MicroPython.

But the challenges with MicroPython are many, including the lack of real documentation on the language. The single page on CPython and MicroPython differences is not really sufficient -- and I am not harping on MicroPython here, the focus on running on different boards drives different documentation needs than the Python on Webassembly use case.

My view is that as long as MicroPython is a desirable target platform due to the fast load times, supporting the deep state would not make it much more shaky. Developer awareness is needed, at least so that they should stick to basics (dict etc.), and if they need to introspect the state object, they might want to untrack it first.

[mikaelho]

On the uses of prune:

If you use the context manager, you might make a number of changes in the different parts of the "tree" formed by the dicts, lists etc. For making the UI update, you might not want to update the whole UI, nor update same parts of the UI several times, so prune gives you the least common paths that have changed.

So if you have an illustrative state like:

• A1
  • A1.1
  • A1.2. <- change here
    • A1.2.1 <- change here

You get changes listed for A1.2 and A1.2.1, but it likely makes sense to just update A1.2 - prune will only give you that path.

[mikaelho]

A little status update on making the introspection work:

• Cannot inherit the proxy object from the class of the object being proxied, due to wrong layout
• Cannot set __instancecheck__ on built-ins like a dict
• Setting __class__ has no impact

For the last one, I looked at MicroPython source code here and compared it to CPython source, and I am not C-worthy enough to start creating a PR. Maybe if @Neon22 could take it up with the MicroPython contacts to see if the assumption that // a.__class__ is equivalent to type(a) could be re-examined?

Meanwhile, isinstance(state.__subject__, dict) is a work-around, at least for PuePy internal code.

[mikaelho]

The version now here also handles the custom objects.

[kkinder]

Nice. Thanks for working on all of this.

In terms of integration, something that occurred to me is that you could define what variables you want deep tracking on in the actual definition of initial(). Something like this:

class MyComponent(Component):
    def initial(self):
        return {
            "project": "Pet Boarding",
            "animals": self.tracked(
                [
                    {"name": "Fido", "species": "dog"},
                    {"name": "Whiskers", "species": "cat"},
                ]
            )
        }

If you call self.tracked, it would keep track of that object deeply. If you don't, it wouldn't.

Thoughts on something like that?

[mikaelho]

Sounds like a nice way to enable whatever level of fine-tuning is needed. 👍

I would personally be more likely to use the kind of blanket control you mentioned earlier: maybe a class variable state_tracking on Component, that could have values of None, "shallow" (default?), "deep", and be customized freely on subclasses or instances.

[kkinder]

The only reason I thought maybe it would be nice to define it inline is that you could specify a depth whether than just have certain keys be reactive. Eg, suppose you have self.state["user"]["profile"] and self.state["user"]["posts"].

You want to react off of user -> profile but not user -> posts. If you set the user key to deep, it'll react off of both. If you set it to shallow, it'll react off of neither. So by calling tracked() on individual items, you can tell PuePy what items you want tracked deeply.

[mikaelho]

This sounds very useful and versatile. My question was mainly about if you do not define anything as "deep", does it mean all the root keys are still tracked? Or that nothing is tracked? Or that everything is tracked? Corresponding to the "shallow"/None/"deep" general setting.

[Neon22]

I posted on Webassembly portion of micropython hoping for some guidance/help here:

• https://github.com/orgs/micropython/discussions/15711

This was because ntoll suggested I did on the Discord forum for pyscript.

[Neon22]

Chris laffra has contributed this to the discussion:

In case this helps, PySheets (which uses PyScript with MicroPython for the UI) uses a Model-View architecture for the cells in a spreadsheet. I override setattr to inform listeners that something changed in the model. This works fine on "standard" PyScript releases for me. See https://github.com/PySheets/pysheets/blob/main/src/static/models.py#L184

[kkinder]

__setattr__ does indeed some to work more or less as expected in MicroPython, and @mikaelho's example makes use of it. The only thing we really miss out on is the ability to subclass dict, list, etc.

[kkinder]

I experimented with this a bit. Something I do wonder is how (if at all), we'd want to handle on_{key}_change. Right now you can just define those automatically, and by virtue of the name, they'll be called as variables change. It's a handy shortcut.

If you have deep keys, how would that work? Maybe not at all?

[mikaelho]

Hmm, maybe not as nice a read, but less prone to surprises: the method names could always be on_state_change (everything), on_state_change_key1_key2 (specific).

[mikaelho]

Good to think if users are allowed or even encouraged to define overlapping handlers, e.g. both on_state_change_user and on_state_change_user_name, and are both called if user name changes, or just the most specific, or the most general?

[mikaelho]

I added here a simple PoC State class that calls the methods named as the latest suggestion above, only calling the most specific callback.

[kkinder]

Hmmm... I think I did something that caused an infinite loop. I wanted to see what happened if you create deeply nested data structure and I added this:

tracked = track({"a": {"b": 1}}, test)
tracked["a"]["b"] = 2
tracked["a"]["c"] = {"x": 1}
tracked["a"]["c"]["y"] = 2

That appears to hang:

https://pyscript.com/@kkinder/reactive-data-test-copy-copy/latest?files=main.py

[kkinder]

As an aside, I'm wondering if thinking too much about how deeply nested on_x_change functions might be unnecessary. If you use track inside the populate method, we could do something like this... It's a bit rough, as I'm just brainstorming here:

class MyComponent(Component):
    def initial(self):
        return {
            # Nested keys are not watched
            "untracked_state": {"key": "value"},

            # Nested keys are watched
            "tracked_state": self.track({
                "key": "value",

                # With a callback
                "nested_cb": self.track({}, on_change=lambda new_value: print("Nested key changed", new_value)),

                # Depth limit
                "nested_not_deep": self.track({}, depth=1),
                
                # Until they get here
                "nested_untracked": self.untrack({})
            }),
        }

So the idea is:

1. You can tell PuePy to track anything, at any depth, or not
2. There's an optional depth parameter that defines how deep state should be tracked
3. There's an additional callback for when state changed for the tracked item

Thus, initial() would really be kind of a shortcut for:

def add_state(self):
    self.track(self.initla(), depth=1)

Something like that?

[kkinder]

Just following up, I want to get this into 0.5.0.

I'm not quite sure how it's best to integrate this into the current codebase and integrate with or replace ReactiveDict, which is pretty foundational to PuePy. I'm still letting it marinade a bit, though admittedly I've been pretty busy the past few weeks with both my "real" job and family stuff.

In the mean time, if you have any ideas, feel free to post them or open a pull request. My general plan thinking is that I want to use a deep parameter to track() and replace ReactiveDict with what you've done.