Missing documentation on Forbidden Relations

[bpkroth]

#245 introduced the very useful feature of constraints between hyperparameters.

However, the documentation at https://automl.github.io/ConfigSpace/main/api/forbidden_clauses.html doesn't seem to include this an instead only lists "values", similar to Conditions.

[JosuaCarl]

Thanks, I was about to point out the same thing :) In the meantime, for all the people looking to restrict hyperparameters, based on the value of another hyperparameter: I would recommend reading the docstrings in https://github.com/automl/ConfigSpace/blob/main/ConfigSpace/forbidden.pyx to learn about the ForbiddenLessThanRelation, ForbiddenEqualsRelation and ForbiddenGreaterThanRelation classes.

[eddiebergman]

Hi @bpkroth and @JosuaCarl,

I've been doing quite a bit of work to refactor ConfigSpace to get rid of Cython while maintaining the speed and reducing technical debt. Extending ConfigSpace has been less than easy and fixing issues neigh impossible.

I'll make sure to add these relations to the documentation. Is there anything else you'd like to see? Please feel free to raise an issue!

Please see this PR.

[bpkroth]

Thanks @eddiebergman that's very exciting to hear. I wonder if there's need/interest for more help in maintainership on some of this stuff going forward?

[bpkroth]

One other thought I'd had related to Forbidden Relations and/or Conditional Relations (which I think don't quite exist yet?): right now there's only basic comparison support. What about adding basic scalar operations to those comparison as well? For instance:

# constraint: buffer_size_mb < vm_size_gb * 1024
ForbiddenLessThanRelation(target_relation="buffer_size_mb", constraint_expr="vm_ram_gb * 1024")

# constraint: buffer_size_1 + buffer_size_2 < vm_size
ForbiddenGreaterThanRelation(target_relation="vm_size", constraint_expr="buffer_size_1 + buffer_size_2")

Happy to open a new Issue on this if you'd prefer.

[eddiebergman]

Thanks @eddiebergman that's very exciting to hear. I wonder if there's need/interest for more help in maintainership on some of this stuff going forward?

@bpkroth Amazing, there definitely would be, however I think this would be after the PR from #346 is done.

I feel like the library will be in much better condition for contribution and we would gladly accept help!

---

One other thought I'd had related to Forbidden Relations and/or Conditional Relations (which I think don't quite exist yet?): right now there's only basic comparison support. What about adding basic scalar operations to those comparison as well? For instance:

# constraint: buffer_size_mb < vm_size_gb * 1024
ForbiddenLessThanRelation(target_relation="buffer_size_mb", constraint_expr="vm_ram_gb * 1024")

# constraint: buffer_size_1 + buffer_size_2 < vm_size
ForbiddenGreaterThanRelation(target_relation="vm_size", constraint_expr="buffer_size_1 + buffer_size_2")

Happy to open a new Issue on this if you'd prefer.

Yes please do open a new issue, I will add this comment on that issue.

I'm pretty certain we'd like to avoid eval expressions of relations. I think what would be much more accessible would be functional forbiddens and conditionals that allow you to perform such operations in a python function.

# Not fleshed out but a rough idea
class MyForbidden(FunctionalForbidden):

    def __init__(self, ...): # Not sure what is required in signature.
        super.__init__(required_things)
        # Set user specific things
        
    # Min required
    def is_valid(self, values: dict[str, Any]) -> None:
        buffer_size = values.get("buffer_size_mb")
        vm_ram_gb = values.get("vm_ram_gb")
        if buffer size is None or vm_ram_gb is None:
        	return True  # Forbiddens are valid if their hyperparameters are not active
        return buffer_size < vm_size_gb * 1024
        
    # Option to @override other methods so they don't rely on
    # converting to the above, i.e. work on vectorized space.
    def is_valid_vector(self, ...) -> None:
    	...

Why it wasn't possible before:

• Probably Cython
• Serialization
• Prevents all sorts of speed optimizations which are important for Bayesian Optimization optimizers (i.e. SMAC) who need to generate thousands of valid random samples and random neighborhood of parameters, quickly, in order to evaluate which configuration would be the next best one to try. This is the original constraint and reason for Cythonizing the code.
  • As a side note, I've managed to make 2/3 of the benchmarked functions 3x faster and the other one is just as fast as the compiled Cython code at this point. I still would like to refactor the tests lastly, as they're pretty hard to read.

Why it's possible now:

• Not Cython
• We just accept that serialization will fail if functional conditionals/forbiddens are provided.
• We just accept that if there are user provided functional conditions/forbiddens, this can be a performance bottleneck for fast target functions being optimized.

[eddiebergman]

The new documentation lists them in the overview reference: https://automl.github.io/ConfigSpace/latest/reference/forbiddens/

However the only real documentation is still API doc: https://automl.github.io/ConfigSpace/latest/api/ConfigSpace/forbidden/#ConfigSpace.forbidden.ForbiddenLessThanRelation

I hope this is explanatory enough but if not, please feel free to let me know what you'd like to see form documentation of it?

[JosuaCarl]

Thanks, at least now it can be found. One thing though, the old documentation can currently no longer be reached. Probably because of the new overall framework, but bold move on a Friday afternoon ;)

[eddiebergman]

Makes my Monday's more exciting ;)

But yes, we can't host the old docs anymore unfortunately as it was a whole different doc setup. However it should be fully backwards compat except for some things mentioned on the homepage