Add Background Compilation for OWLv2 Vision Model

[lrosemberg]

Description

Added a singleton manager to handle OWLv2's vision model compilation in background, avoiding the initial 2-minute blocking compilation while maintaining the performance benefits of compiled models.

Changes

• Added OWLv2ModelManager class that manages vision model compilation per huggingface_id
• Compilation happens in a daemon thread, allowing immediate model usage
• Uses thread-safe singleton pattern to ensure one compiled model instance per huggingface_id
• Integrated with existing Owlv2Singleton class
• No changes to the model's API or behavior

Example Usage

model = OwlV2(model_id=f"owlv2/{OWLV2_VERSION_ID}")
# Model is immediately usable, compilation happens in background
response = model.infer_from_request(request)

Benefits

• Eliminates 2-minute blocking during model initialization
• Maintains compiled model performance benefits
• Thread-safe
• Transparent to existing code using the model

Potential Risks

• First inference might be slightly slower (using uncompiled model)
• Daemon threads might be killed during program termination (acceptable as compilation is non-critical)

Type of change

• New feature (non-breaking change which adds functionality)

How has this change been tested, please provide a testcase or example of how you tested the change?

Added tests to verify:

• Singleton behavior per huggingface_id
• Proper garbage collection of manager instances

Any specific deployment considerations

• Deploy inference internal

[CLAassistant]

All committers have signed the CLA.

[lrosemberg]

Hello @PawelPeczek-Roboflow, In addition to local testing, it was also deployed to the inference-internal of staging and production, and solved the problems we had with the long running in inference for instant models, which previously caused an error because the timeout on the nginx side of the inference-internal was 30s, now it runs around 3s.

That's a smarter way to do the previous COMPILE_OWLV2_MODEL thing. Instead of not compiling and always having slower inferences, the first time we return the non-compiled instance and start the compilation process in the background and after compiling we replace the instance, making the next inferences faster.

Slack threads for context: results, discussion thread