Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs][PyOV] Documentation for Custom Python Operation #25615

Merged
merged 34 commits into from
Feb 3, 2025
Merged
Show file tree
Hide file tree
Changes from 29 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
205044d
[Docs][PyOV] Documentation for Custom Python Operation
akuporos Jul 17, 2024
13c0a10
make the snippet work
akuporos Jul 18, 2024
5b7f90e
update dpc
akuporos Jul 18, 2024
e0f40f9
try to fix doc build
akuporos Jul 18, 2024
8a417a6
Merge branch 'master' into akup/custom-op-docs
akuporos Jul 22, 2024
95ba862
change a little bit
akuporos Jul 22, 2024
091af42
Update docs/articles_en/documentation/openvino-extensibility/custom-o…
akuporos Jul 22, 2024
3cb000a
more changes
akuporos Jul 22, 2024
9939776
Update docs/articles_en/documentation/openvino-extensibility.rst
akuporos Jul 22, 2024
f0c3733
Merge branch 'master' into akup/custom-op-docs
mlukasze Jul 23, 2024
4e20847
Merge branch 'master' into akup/custom-op-docs
akuporos Jul 30, 2024
a237a03
Update docs/articles_en/documentation/openvino-extensibility.rst
akuporos Jul 30, 2024
753562d
Update docs/articles_en/documentation/openvino-extensibility.rst
akuporos Jul 30, 2024
9ba1424
Update docs/articles_en/documentation/openvino-extensibility.rst
akuporos Jul 30, 2024
d25ac84
Update docs/articles_en/documentation/openvino-extensibility/custom-o…
akuporos Jul 30, 2024
1c4732f
apply comment
akuporos Jul 31, 2024
110fd42
Merge branch 'master' into akup/custom-op-docs
akuporos Sep 6, 2024
c4d1de8
Update docs/articles_en/documentation/openvino-extensibility/custom-o…
kblaszczak-intel Sep 12, 2024
815bba4
Merge branch 'master' into akup/custom-op-docs
akuporos Sep 13, 2024
32d9076
Merge branch 'master' into akup/custom-op-docs
akuporos Sep 28, 2024
0d0543f
Merge branch 'master' into akup/custom-op-docs
akuporos Oct 14, 2024
b505644
Merge branch 'master' into akup/custom-op-docs
kblaszczak-intel Oct 28, 2024
77040b4
Merge branch 'master' into akup/custom-op-docs
akuporos Jan 16, 2025
ec254d7
some updates
akuporos Jan 23, 2025
27673d0
Merge branch 'master' into akup/custom-op-docs
mlukasze Jan 24, 2025
c3ceabc
update custom-op-doc
akuporos Jan 28, 2025
f051ce1
update extensions
akuporos Jan 28, 2025
7ca4439
apply comments
akuporos Jan 28, 2025
bc26774
Merge branch 'master' into akup/custom-op-docs
akuporos Jan 29, 2025
240c23f
small update
akuporos Jan 29, 2025
7ecd86d
Update docs/articles_en/documentation/openvino-extensibility/custom-o…
akuporos Jan 29, 2025
4b76a4c
Apply suggestions from code review
sgolebiewski-intel Feb 3, 2025
3c33e61
Apply suggestions from code review
sgolebiewski-intel Feb 3, 2025
1aaeca4
Apply suggestions from code review
sgolebiewski-intel Feb 3, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions docs/articles_en/assets/snippets/ov_custom_op.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Copyright (C) 2018-2025 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
#

# ! [op:common_include]
from openvino import Op
# ! [op:common_include]



# ! [op:header]
class Identity(Op):
# ! [op:header]

# ! [op:ctor]
def __init__(self, inputs=None, **attrs):
super().__init__(self, inputs)
self._attrs = attrs
# ! [op:ctor]

# ! [op:validate]
def validate_and_infer_types(self):
self.set_output_type(0, self.get_input_element_type(0), self.get_input_partial_shape(0))
# ! [op:validate]

# ! [op:copy]
def clone_with_new_inputs(self, new_inputs):
return Identity(new_inputs)
# ! [op:copy]

# ! [op:evaluate]
def evaluate(self, outputs, inputs):
outputs[0].shape = inputs[0].shape
mlukasze marked this conversation as resolved.
Show resolved Hide resolved
inputs[0].copy_to(outputs[0])
return True

def has_evaluate(self):
return True
Comment on lines +37 to +38
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we avoid doing that by just defining evaluate?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I understand this method is obligatory and always goes together with evaluate

# ! [op:evaluate]

# ! [op:visit_attributes]
def visit_attributes(self, visitor):
visitor.on_attributes(self._attrs)
return True
# ! [op:visit_attributes]
18 changes: 15 additions & 3 deletions docs/articles_en/assets/snippets/ov_extensions.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,30 @@
#

import openvino as ov
from ov_custom_op import Identity

#! [py_frontend_extension_ThresholdedReLU_header]
import openvino.runtime.opset12 as ops
import openvino.runtime.opset14 as ops
from openvino.frontend import ConversionExtension
#! [py_frontend_extension_ThresholdedReLU_header]

#! [add_extension]
# Not implemented
core = ov.Core()

# Use operation type to add operation extension
core.add_extension(Identity)

# or you can add operation extension object which is equivalent form
core.add_extension(ov.OpExtension(Identity))
#! [add_extension]

#! [add_frontend_extension]
# Not implemented
# Register more sophisticated mapping with decomposition
def conversion(node):
input_node = node.get_input(0)
return Identity(input_node).outputs()

core.add_extension(ConversionExtension("Identity", conversion))
#! [add_frontend_extension]

from utils import get_path_to_extension_library
Expand Down
6 changes: 3 additions & 3 deletions docs/articles_en/documentation/openvino-extensibility.rst
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,8 @@ If such decomposition is not possible or appears too bulky with a large number o

You might prefer implementing a custom operation class if you already have a generic C++ implementation of operation kernel. Otherwise, try to decompose the operation first, as described above. Then, after verifying correctness of inference and resulting performance, you may move on to optional implementation of Bare Metal C++.

Additionally, it is also possible to implement custom operations using Python. OpenVINO provides a Python API that allows you to define and register custom operations. This can be particularly useful for rapid prototyping and testing of new operations.

Mapping from Framework Operation
################################

Expand Down Expand Up @@ -110,9 +112,7 @@ The ``Identity`` is a custom operation class defined in :doc:`Custom Operation G
:language: cpp
:fragment: [add_frontend_extension]

When Python API is used, there is no way to implement a custom OpenVINO operation. Even if custom OpenVINO operation is implemented in C++ and loaded into the runtime by a shared library, there is still no way to add a frontend mapping extension that refers to this custom operation. In this case, use C++ shared library approach to implement both operations semantics and framework mapping.

akuporos marked this conversation as resolved.
Show resolved Hide resolved
Python can still be used to map and decompose operations when only operations from the standard OpenVINO operation set are used.
If custom OpenVINO operation is implemented in C++ and loaded into the runtime using Python API by a shared library, there is no way to add a frontend mapping extension that refers to this custom operation. In this case, use C++ shared library approach to implement both operations semantics and framework mapping.
akuporos marked this conversation as resolved.
Show resolved Hide resolved

.. _create_a_library_with_extensions:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,30 +7,62 @@ Custom OpenVINO Operations
custom operations to support models with operations
not supported by OpenVINO.

OpenVINO™ Extension API allows you to register custom operations to support models with operations which OpenVINO™ does not support out-of-the-box. This capability requires writing code in C++, so if you are using Python to develop your application you need to build a separate shared library implemented in C++ first and load it in Python using ``add_extension`` API. Please refer to :ref:`Create library with extensions <create_a_library_with_extensions>` for more details on library creation and usage. The remaining part of this document describes how to implement an operation class.
OpenVINO™ Extension API allows you to register custom operations to support models with operations which OpenVINO™ does not support out-of-the-box. A custom operation might be implemented both in C++ and Python.```

Also it is possible to create a shared library with custom operation implemented in C++ first and load it using ``add_extension`` API. Please refer to :ref:`Create library with extensions <create_a_library_with_extensions>` for more details on library creation and usage. The remaining part of this document describes how to implement an operation class using both the C++ API and Python API.

Operation Class
###############

To add your custom operation, create a new class that extends ``ov::Op``, which is in turn derived from ``ov::Node``, the base class for all graph operations in OpenVINO™. To add ``ov::Op``, include the next file:
.. tab-set::

.. tab-item:: C++
sgolebiewski-intel marked this conversation as resolved.
Show resolved Hide resolved

sgolebiewski-intel marked this conversation as resolved.
Show resolved Hide resolved
To add your custom operation, create a new class that extends ``ov::Op``, which is in turn derived from ``ov::Node``, the base class for all graph operations in OpenVINO™. To add ``ov::Op``, include the next file:
sgolebiewski-intel marked this conversation as resolved.
Show resolved Hide resolved

:sync: cpp
sgolebiewski-intel marked this conversation as resolved.
Show resolved Hide resolved

.. doxygensnippet:: src/core/template_extension/identity.hpp
:language: cpp
:fragment: [op:common_include]

Follow the steps below to add a custom operation:

1. Add the ``OPENVINO_OP`` macro. The type info of an operation consists of a string operation identifier and a string for operation version.

2. Implement default constructor and constructors that optionally take the operation inputs and attributes as parameters.

3. Override the shape inference method ``validate_and_infer_types``. This method is called multiple times during graph manipulations to determine the shapes and element types of the operations outputs. To access the input shapes and input element types, use the ``get_input_partial_shape()`` and ``get_input_element_type()`` methods of ``ov::Node``. Set the inferred shape and element type of the output using ``set_output_type``.

4. Override the ``clone_with_new_inputs`` method, which enables graph manipulation routines to create copies of this operation and connect it to different nodes during optimization.

5. Override the ``visit_attributes`` method, which enables serialization and deserialization of operation attributes. An ``AttributeVisitor`` is passed to the method, and the implementation is expected to walk over all the attributes in the op using the type-aware ``on_attribute`` helper. Helpers are already implemented for standard C++ types like ``int64_t``, ``float``, ``bool``, ``vector``, and for existing OpenVINO defined types.

6. Override ``evaluate`` method, which enables fallback of some devices to this implementation and the application of constant folding if there is a custom operation on the constant branch. If your operation contains ``evaluate`` method you also need to override the ``has_evaluate`` method, this method allows to get information about availability of ``evaluate`` method for the operation.


.. tab-item:: Python

sgolebiewski-intel marked this conversation as resolved.
Show resolved Hide resolved
.. doxygensnippet:: src/core/template_extension/identity.hpp
:language: cpp
:fragment: [op:common_include]
To add your custom operation, create a new class that extends ``openvino.Op``, which is in turn derived from ``openvino.Node``, the base class for all graph operations in OpenVINO™. To add ``openvino.Op`` you need to import it.

Follow the steps below to add a custom operation:
:sync: py
sgolebiewski-intel marked this conversation as resolved.
Show resolved Hide resolved

1. Add the ``OPENVINO_OP`` macro which defines a ``NodeTypeInfo`` object that identifies the type of the operation to the graph users and helps with dynamic type resolution. The type info of an operation currently consists of a string operation identifier and a string for operation version.
.. doxygensnippet:: docs/articles_en/assets/snippets/ov_custom_op.py
:language: python
:fragment: [op:common_include]

2. Implement default constructor and constructors that optionally take the operation inputs and attributes as parameters.
Follow the steps below to add a simple custom operation:

3. Override the shape inference method ``validate_and_infer_types``. This method is called multiple times during graph manipulations to determine the shapes and element types of the operations outputs. To access the input shapes and input element types, use the ``get_input_partial_shape()`` and ``get_input_element_type()`` methods of ``ov::Node``. Set the inferred shape and element type of the output using ``set_output_type``.
1. Define the ``__init__`` method to initialize the class with inputs and attributes.

4. Override the ``clone_with_new_inputs`` method, which enables graph manipulation routines to create copies of this operation and connect it to different nodes during optimization.
2. Override the shape inference method ``validate_and_infer_types``. This method is called multiple times during graph manipulations to determine the shapes and element types of the operations outputs. To access the input shapes and input element types, use the ``get_input_partial_shape()`` and ``get_input_element_type()`` methods of ``openvino.Node``. Set the inferred shape and element type of the output using ``set_output_type``.

5. Override the ``visit_attributes`` method, which enables serialization and deserialization of operation attributes. An ``AttributeVisitor`` is passed to the method, and the implementation is expected to walk over all the attributes in the op using the type-aware ``on_attribute`` helper. Helpers are already implemented for standard C++ types like ``int64_t``, ``float``, ``bool``, ``vector``, and for existing OpenVINO defined types.
3. Override the ``visit_attributes`` method, which enables serialization and deserialization of operation attributes. An ``AttributeVisitor`` is passed to the method, and the implementation is expected to walk over all the attributes in the op using the type-aware ``on_attribute`` helper. Helpers are already implemented for standard types like ``int``, ``float``, ``bool``, ``vector``, and for existing OpenVINO defined types.

4. Override ``evaluate`` method, which enables fallback of a CPU device to this implementation and the application of constant folding if there is a custom operation on the constant branch. If your operation contains ``evaluate`` method you also need to override the ``has_evaluate`` method, this method allows to get information about availability of ``evaluate`` method for the operation.
akuporos marked this conversation as resolved.
Show resolved Hide resolved

5. Override the ``clone_with_new_inputs`` method, which enables graph manipulation routines to create copies of this operation and connect it to different nodes during optimization.

sgolebiewski-intel marked this conversation as resolved.
Show resolved Hide resolved
6. Override ``evaluate``, which is an optional method that enables fallback of some devices to this implementation and the application of constant folding if there is a custom operation on the constant branch. If your operation contains ``evaluate`` method you also need to override the ``has_evaluate`` method, this method allows to get information about availability of ``evaluate`` method for the operation.

Based on that, declaration of an operation class can look as follows:

Expand All @@ -43,43 +75,103 @@ OpenVINO™ operation contains two constructors:
* Default constructor, which enables you to create an operation without attributes
* Constructor that creates and validates an operation with specified inputs and attributes

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:ctor]
.. tab-set::

.. tab-item:: C++
:sync: cpp

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:ctor]

.. tab-item:: Python
:sync: py

.. doxygensnippet:: docs/articles_en/assets/snippets/ov_custom_op.py
:language: python
:fragment: [op:ctor]
sgolebiewski-intel marked this conversation as resolved.
Show resolved Hide resolved

``validate_and_infer_types()``
++++++++++++++++++++++++++++++

``ov::Node::validate_and_infer_types`` method validates operation attributes and calculates output shapes using attributes of the operation.

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:validate]
.. tab-set::

.. tab-item:: C++
:sync: cpp

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:validate]

.. tab-item:: Python
:sync: py

.. doxygensnippet:: docs/articles_en/assets/snippets/ov_custom_op.py
:language: python
:fragment: [op:validate]
sgolebiewski-intel marked this conversation as resolved.
Show resolved Hide resolved

``clone_with_new_inputs()``
+++++++++++++++++++++++++++

``ov::Node::clone_with_new_inputs`` method creates a copy of the operation with new inputs.

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:copy]
.. tab-set::

.. tab-item:: C++
:sync: cpp

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:copy]

.. tab-item:: Python
:sync: py

.. doxygensnippet:: docs/articles_en/assets/snippets/ov_custom_op.py
:language: python
:fragment: [op:copy]
sgolebiewski-intel marked this conversation as resolved.
Show resolved Hide resolved

``visit_attributes()``
++++++++++++++++++++++

``ov::Node::visit_attributes`` method enables you to visit all operation attributes.

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:visit_attributes]
.. tab-set::

.. tab-item:: C++
:sync: cpp

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:visit_attributes]

.. tab-item:: Python
:sync: py

.. doxygensnippet:: docs/articles_en/assets/snippets/ov_custom_op.py
:language: python
:fragment: [op:visit_attributes]
sgolebiewski-intel marked this conversation as resolved.
Show resolved Hide resolved

``evaluate() and has_evaluate()``
+++++++++++++++++++++++++++++++++

``ov::Node::evaluate`` method enables you to apply constant folding to an operation.

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:evaluate]
.. tab-set::

.. tab-item:: C++
:sync: cpp

.. doxygensnippet:: src/core/template_extension/identity.cpp
:language: cpp
:fragment: [op:evaluate]

.. tab-item:: Python
:sync: py

.. doxygensnippet:: docs/articles_en/assets/snippets/ov_custom_op.py
:language: python
:fragment: [op:evaluate]
sgolebiewski-intel marked this conversation as resolved.
Show resolved Hide resolved

Loading