Add TensorTree type

[SamDuffield]

Added TensorTree type. Don't think there are other custom types we'd like to add currently? But I think there will be in future.

[dmitrisaberi]

So PyTreeTypeVar("custom_name", Type) defines a type custom_name of pytree where the leaf nodes all have type Type?

[SamDuffield]

Yeah see optree.PyTreeTypeVar

Although looking at it now PyTree[torch.Tensor] is perhaps cleaner

[dmitrisaberi]

i think either is fine :)

[SamDuffield]

I'll keep like this for now, this is what torchopt does 😄

[dmitrisaberi]

this may be a stupid question but why do we care so much about generalizing to tree-like structures? what are some crucial examples where parameters are not just dicts? (otherwise LGTM)

[SamDuffield]

this may be a stupid question but why do we care so much about generalizing to tree-like structures? what are some crucial examples where parameters are not just dicts? (otherwise LGTM)

The object to be optimised could just be a tensor (useful for testing and more traditional MCMC).

Also can support lists of dicts (which I think is common in pytorch with so-called "parameter groups") https://pytorch.org/docs/stable/optim.html

[dmitrisaberi]

this may be a stupid question but why do we care so much about generalizing to tree-like structures? what are some crucial examples where parameters are not just dicts? (otherwise LGTM)

The object to be optimised could just be a tensor (useful for testing and more traditional MCMC).

Also can support lists of dicts (which I think is common in pytorch with so-called "parameter groups") https://pytorch.org/docs/stable/optim.html

makes sense, thanks!