diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..bea32dc --- /dev/null +++ b/LICENSE @@ -0,0 +1,130 @@ +1. Copyright & License Notices + + +1.1. NVIDIA CORPORATION +1.1.1. NVIDIA SOFTWARE LICENSE AGREEMENT +IMPORTANT – READ BEFORE DOWNLOADING, INSTALLING, COPYING OR USING THE LICENSED SOFTWARE + +This Software License Agreement ("SLA”), made and entered into as of the time and date of click through action (“Effective Date”), is a legal agreement between you and NVIDIA Corporation ("NVIDIA") and governs the use of the NVIDIA computer software and the documentation made available for use with such NVIDIA software. By downloading, installing, copying, or otherwise using the NVIDIA software and/or documentation, you agree to be bound by the terms of this SLA. If you do not agree to the terms of this SLA, do not download, install, copy or use the NVIDIA software or documentation. IF YOU ARE ENTERING INTO THIS SLA ON BEHALF OF A COMPANY OR OTHER LEGAL ENTITY, YOU REPRESENT THAT YOU HAVE THE LEGAL AUTHORITY TO BIND THE ENTITY TO THIS SLA, IN WHICH CASE “YOU” WILL MEAN THE ENTITY YOU REPRESENT. IF YOU DON’T HAVE SUCH AUTHORITY, OR IF YOU DON’T ACCEPT ALL THE TERMS AND CONDITIONS OF THIS SLA, THEN NVIDIA DOES NOT AGREE TO LICENSE THE LICENSED SOFTWARE TO YOU, AND YOU MAY NOT DOWNLOAD, INSTALL, COPY OR USE IT. + +1. LICENSE. + +1.1 License Grant. Subject to the terms of the AGREEMENT, NVIDIA hereby grants you a non-exclusive, non-transferable license, without the right to sublicense (except as expressly set forth in a Supplement), during the applicable license term unless earlier terminated as provided below, to have Authorized Users install and use the Software, including modifications (if expressly permitted in a Supplement), in accordance with the Documentation. You are only licensed to activate and use Licensed Software for which you a have a valid license, even if during the download or installation you are presented with other product options. No Orders are binding on NVIDIA until accepted by NVIDIA. Your Orders are subject to the AGREEMENT. + +SLA Supplements: Certain Licensed Software licensed under this SLA may be subject to additional terms and conditions that will be presented to you in a Supplement for acceptance prior to the delivery of such Licensed Software under this SLA and the applicable Supplement. Licensed Software will only be delivered to you upon your acceptance of all applicable terms. + +1.2 Limited Purpose Licenses. If your license is provided for one of the purposes indicated below, then notwithstanding contrary terms in Section 1.1 or in a Supplement, such licenses are for internal use and do not include any right or license to sub-license and distribute the Licensed Software or its output in any way in any public release, however limited, and/or in any manner that provides third parties with use of or access to the Licensed Software or its functionality or output, including (but not limited to) external alpha or beta testing or development phases. Further: + +(i) Evaluation License. You may use evaluation licenses solely for your internal evaluation of the Licensed Software for broader adoption within your Enterprise or in connection with a NVIDIA product purchase decision, and such licenses have an expiration date as indicated by NVIDIA in its sole discretion (or ninety days from the date of download if no other duration is indicated). + +(ii) Educational/Academic License. You may use educational/academic licenses solely for educational purposes and all users must be enrolled or employed by an academic institution. If you do not meet NVIDIA’s academic program requirements for educational institutions, you have no rights under this license. + +(iii) Test/Development License. You may use test/development licenses solely for your internal development, testing and/or debugging of your software applications or for interoperability testing with the Licensed Software, and such licenses have an expiration date as indicated by NVIDIA in its sole discretion (or one year from the date of download if no other duration is indicated). + +1.3 Pre-Release Licenses. With respect to alpha, beta, preview, and other pre-release Software and Documentation (“Pre-Release Licensed Software”) delivered to you under the AGREEMENT you acknowledge and agree that such Pre-Release Licensed Software (i) may not be fully functional, may contain errors or design flaws, and may have reduced or different security, privacy, accessibility, availability, and reliability standards relative to commercially provided NVIDIA software and documentation, and (ii) use of such Pre-Release Licensed Software may result in unexpected results, loss of data, project delays or other unpredictable damage or loss. THEREFORE, PRE-RELEASE LICENSED SOFTWARE IS NOT INTENDED FOR USE, AND SHOULD NOT BE USED, IN PRODUCTION OR BUSINESS-CRITICAL SYSTEMS. NVIDIA has no obligation to make available a commercial version of any Pre-Release Licensed Software and NVIDIA has the right to abandon development of Pre-Release Licensed Software at any time without liability. + +1.4 Enterprise and Contractor Usage. You may allow your Enterprise employees and Contractors to access and use the Licensed Software pursuant to the terms of the AGREEMENT solely to perform work on your behalf, provided further that with respect to Contractors: (i) you obtain a written agreement from each Contractor which contains terms and obligations with respect to access to and use of Licensed Software no less protective of NVIDIA than those set forth in the AGREEMENT, and (ii) such Contractor’s access and use expressly excludes any sublicensing or distribution rights for the Licensed Software. You are responsible for the compliance with the terms and conditions of the AGREEMENT by your Enterprise and Contractors. Any act or omission that, if committed by you, would constitute a breach of the AGREEMENT shall be deemed to constitute a breach of the AGREEMENT if committed by your Enterprise or Contractors. + +1.5 Services. Except as expressly indicated in an Order, NVIDIA is under no obligation to provide support for the Licensed Software or to provide any patches, maintenance, updates or upgrades under the AGREEMENT. Unless patches, maintenance, updates or upgrades are provided with their separate governing terms and conditions, they constitute Licensed Software licensed to you under the AGREEMENT. + +2. LIMITATIONS. + +2.1 License Restrictions. Except as expressly authorized in the AGREEMENT, you agree that you will not (nor authorize third parties to): (i) copy and use Software that was licensed to you for use in one or more NVIDIA hardware products in other unlicensed products (provided that copies solely for backup purposes are allowed); (ii) reverse engineer, decompile, disassemble (except to the extent applicable laws specifically require that such activities be permitted) or attempt to derive the source code, underlying ideas, algorithm or structure of Software provided to you in object code form; (iii) sell, transfer, assign, distribute, rent, loan, lease, sublicense or otherwise make available the Licensed Software or its functionality to third parties (a) as an application services provider or service bureau, (b) by operating hosted/virtual system environments, (c) by hosting, time sharing or providing any other type of services, or (d) otherwise by means of the internet; (iv) modify, translate or otherwise create any derivative works of any Licensed Software; (v) remove, alter, cover or obscure any proprietary notice that appears on or with the Licensed Software or any copies thereof; (vi) use the Licensed Software, or allow its use, transfer, transmission or export in violation of any applicable export control laws, rules or regulations; (vii) distribute, permit access to, or sublicense the Licensed Software as a stand-alone product; (viii) bypass, disable, circumvent or remove any form of copy protection, encryption, security or digital rights management or authentication mechanism used by NVIDIA in connection with the Licensed Software, or use the Licensed Software together with any authorization code, serial number, or other copy protection device not supplied by NVIDIA directly or through an authorized reseller; (ix) use the Licensed Software for the purpose of developing competing products or technologies or assisting a third party in such activities; (x) use the Licensed Software with any system or application where the use or failure of such system or application can reasonably be expected to threaten or result in personal injury, death, or catastrophic loss including, without limitation, use in connection with any nuclear, avionics, navigation, military, medical, life support or other life critical application (“Critical Applications”), unless the parties have entered into a Critical Applications agreement; (xi) distribute any modification or derivative work you make to the Licensed Software under or by reference to the same name as used by NVIDIA; or (xii) use the Licensed Software in any manner that would cause the Licensed Software to become subject to an Excluded License. Nothing in the AGREEMENT shall be construed to give you a right to use, or otherwise obtain access to, any source code from which the Software or any portion thereof is compiled or interpreted. You acknowledge that NVIDIA does not design, test, manufacture or certify the Licensed Software for use in the context of a Critical Application and NVIDIA shall not be liable to you or any third party, in whole or in part, for any claims or damages arising from such use. You agree to defend, indemnify and hold harmless NVIDIA and its Affiliates, and their respective employees, contractors, agents, officers and directors, from and against any and all claims, damages, obligations, losses, liabilities, costs or debt, fines, restitutions and expenses (including but not limited to attorney’s fees and costs incident to establishing the right of indemnification) arising out of or related to you and your Enterprise, and their respective employees, contractors, agents, distributors, resellers, end users, officers and directors use of Licensed Software outside of the scope of the AGREEMENT or any other breach of the terms of the AGREEMENT. + +2.2 Third Party License Obligations. The Licensed Software may come bundled with, or otherwise include or be distributed with, third party software licensed by an NVIDIA supplier and/or open source software provided under an open source license (collectively, “Third Party Software”). Notwithstanding anything to the contrary herein, Third Party Software is licensed to you subject to the terms and conditions of the software license agreement accompanying such Third Party Software whether in the form of a discrete agreement, click-through license, or electronic license terms accepted at the time of installation and any additional terms or agreements provided by the third party licensor (“Third Party License Terms”). Use of the Third Party Software by you shall be governed by such Third Party License Terms, or if no Third Party License Terms apply, then the Third Party Software is provided to you as-is, without support or warranty or indemnity obligations, for use in or with the Licensed Software and not otherwise used separately. Copyright to Third Party Software is held by the copyright holders indicated in the Third Party License Terms. + +Audio/Video Encoders and Decoders. You acknowledge and agree that it is your sole responsibility to obtain any additional third party licenses required to make, have made, use, have used, sell, import, and offer for sale your products or services that include or incorporate any Third Party Software and content relating to audio and/or video encoders and decoders from, including but not limited to, Microsoft, Thomson, Fraunhofer IIS, Sisvel S.p.A., MPEG-LA, and Coding Technologies as NVIDIA does not grant to you under the AGREEMENT any necessary patent or other rights with respect to audio and/or video encoders and decoders. + +2.3 Limited Rights. Your rights in the Licensed Software are limited to those expressly granted under the AGREEMENT and no other licenses are granted whether by implication, estoppel or otherwise. NVIDIA reserves all rights, title and interest in and to the Licensed Software not expressly granted under the AGREEMENT. + +3. CONFIDENTIALITY. Neither party will use the other party’s Confidential Information, except as necessary for the performance of the AGREEMENT, nor will either party disclose such Confidential Information to any third party, except to personnel of NVIDIA and its Affiliates, you, your Enterprise, your Enterprise Contractors, and each party’s legal and financial advisors that have a need to know such Confidential Information for the performance of the AGREEMENT, provided that each such personnel, employee and Contractors are subject to a written agreement that includes confidentiality obligations consistent with those set forth herein. Each party will use all reasonable efforts to maintain the confidentiality of all of the other party’s Confidential Information in its possession or control, but in no event less than the efforts that it ordinarily uses with respect to its own Confidential Information of similar nature and importance. The foregoing obligations will not restrict either party from disclosing the other party’s Confidential Information or the terms and conditions of the AGREEMENT as required under applicable securities regulations or pursuant to the order or requirement of a court, administrative agency, or other governmental body, provided that the party required to make such disclosure (i) gives reasonable notice to the other party to enable it to contest such order or requirement prior to its disclosure (whether through protective orders or otherwise), (ii) uses reasonable effort to obtain confidential treatment or similar protection to the fullest extent possible to avoid such public disclosure, and (iii) discloses only the minimum amount of information necessary to comply with such requirements. + +NVIDIA Confidential Information under the AGREEMENT includes output from Licensed Software developer tools identified as “Pro” versions, where the output reveals functionality or performance data pertinent to NVIDIA hardware or software products. + +4. OWNERSHIP. You are not obligated to disclose to NVIDIA any modifications that you, your Enterprise or your Contractors make to the Licensed Software as permitted under the AGREEMENT. As between the parties, all modifications are owned by NVIDIA and licensed to you under the AGREEMENT unless otherwise expressly provided in a Supplement. The Licensed Software and all modifications owned by NVIDIA, and the respective Intellectual Property Rights therein, are and will remain the sole and exclusive property of NVIDIA or its licensors. You shall not engage in any act or omission that would impair NVIDIA’s and/or its licensors’ Intellectual Property Rights in the Licensed Software or any other materials, information, processes or subject matter proprietary to NVIDIA. NVIDIA’s licensors are intended third party beneficiaries with the right to enforce provisions of the AGREEMENT with respect to their Confidential Information and/or Intellectual Property Rights. + +5. FEEDBACK. You may, but you are not obligated, to provide Feedback to NVIDIA. You hereby grant NVIDIA and its Affiliates a perpetual, non-exclusive, worldwide, irrevocable license to use, reproduce, modify, license, sublicense (through multiple tiers of sublicensees), distribute (through multiple tiers of distributors) and otherwise commercialize any Feedback that you voluntarily provide without the payment of any royalties or fees to you. NVIDIA has no obligation to respond to Feedback or to incorporate Feedback into the Licensed Software. + +6. NO WARRANTIES. THE LICENSED SOFTWARE AND ANY CONFIDENTIAL INFORMATION AND/OR SERVICES ARE PROVIDED BY NVIDIA “AS IS” AND “WITH ALL FAULTS,” AND NVIDIA AND ITS AFFILIATES EXPRESSLY DISCLAIM ALL WARRANTIES OF ANY KIND OR NATURE, WHETHER EXPRESS, IMPLIED OR STATUTORY, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTIES OF OPERABILITY, CONDITION, VALUE, ACCURACY OF DATA, OR QUALITY, AS WELL AS ANY WARRANTIES OF MERCHANTABILITY, SYSTEM INTEGRATION, WORKMANSHIP, SUITABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE, NON-INFRINGEMENT, OR THE ABSENCE OF ANY DEFECTS THEREIN, WHETHER LATENT OR PATENT. NO WARRANTY IS MADE ON THE BASIS OF TRADE USAGE, COURSE OF DEALING OR COURSE OF TRADE. WITHOUT LIMITING THE FOREGOING, NVIDIA AND ITS AFFILIATES DO NOT WARRANT THAT THE LICENSED SOFTWARE OR ANY CONFIDENTIAL INFORMATION AND/OR SERVICES PROVIDED UNDER THE AGREEMENT WILL MEET YOUR REQUIREMENTS OR THAT THE OPERATION THEREOF WILL BE UNINTERRUPTED OR ERROR-FREE, OR THAT ALL ERRORS WILL BE CORRECTED. + +7. LIMITATION OF LIABILITY. TO THE MAXIMUM EXTENT PERMITTED BY LAW, NVIDIA AND ITS AFFILIATES SHALL NOT BE LIABLE FOR ANY SPECIAL, INCIDENTAL, PUNITIVE OR CONSEQUENTIAL DAMAGES, OR ANY LOST PROFITS, LOSS OF USE, LOSS OF DATA OR LOSS OF GOODWILL, OR THE COSTS OF PROCURING SUBSTITUTE PRODUCTS, ARISING OUT OF OR IN CONNECTION WITH THE AGREEMENT OR THE USE OR PERFORMANCE OF THE LICENSED SOFTWARE AND ANY CONFIDENTIAL INFORMATION AND/OR SERVICES PROVIDED UNDER THE AGREEMENT, WHETHER SUCH LIABILITY ARISES FROM ANY CLAIM BASED UPON BREACH OF CONTRACT, BREACH OF WARRANTY, TORT (INCLUDING NEGLIGENCE), PRODUCT LIABILITY OR ANY OTHER CAUSE OF ACTION OR THEORY OF LIABILITY. IN NO EVENT WILL NVIDIA’S AND ITS AFFILIATES TOTAL CUMULATIVE LIABILITY UNDER OR ARISING OUT OF THE AGREEMENT EXCEED THE NET AMOUNTS RECEIVED BY NVIDIA OR ITS AFFILIATES FOR YOUR USE OF THE PARTICULAR LICENSED SOFTWARE DURING THE TWELVE (12) MONTHS BEFORE THE LIABILITY AROSE (or up to US$10.00 if you acquired the Licensed Software for no charge). THE NATURE OF THE LIABILITY, THE NUMBER OF CLAIMS OR SUITS OR THE NUMBER OF PARTIES WITHIN YOUR ENTERPRISE THAT ACCEPTED THE TERMS OF THE AGREEMENT SHALL NOT ENLARGE OR EXTEND THIS LIMIT. THE FOREGOING LIMITATIONS SHALL APPLY REGARDLESS OF WHETHER NVIDIA, ITS AFFILIATES OR ITS LICENSORS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES AND REGARDLESS OF WHETHER ANY REMEDY FAILS ITS ESSENTIAL PURPOSE. YOU ACKNOWLEDGE THAT NVIDIA’S OBLIGATIONS UNDER THE AGREEMENT ARE FOR THE BENEFIT OF YOU ONLY. The disclaimers, exclusions and limitations of liability set forth in the AGREEMENT form an essential basis of the bargain between the parties, and, absent any such disclaimers, exclusions or limitations of liability, the provisions of the AGREEMENT, including, without limitation, the economic terms, would be substantially different. + +8. TERM AND TERMINATION. + +8.1 AGREEMENT, Licenses and Services. This SLA shall become effective upon the Effective Date, each Supplement upon their acceptance, and both this SLA and Supplements shall continue in effect until your last access or use of the Licensed Software and/or services hereunder, unless earlier terminated as provided in this “Term and Termination” section. Each Licensed Software license ends at the earlier of (a) the expiration of the applicable license term, or (b) termination of such license or the AGREEMENT. Each service ends at the earlier of (x) the expiration of the applicable service term, (y) termination of such service or the AGREEMENT, or (z) expiration or termination of the associated license and no credit or refund will be provided upon the expiration or termination of the associated license for any service fees paid. + +8.2 Termination and Effect of Expiration or Termination. NVIDIA may terminate the AGREEMENT in whole or in part: (i) if you breach any term of the AGREEMENT and fail to cure such breach within thirty (30) days following notice thereof from NVIDIA (or immediately if you violate NVIDIA’s Intellectual Property Rights); (ii) if you become the subject of a voluntary or involuntary petition in bankruptcy or any proceeding relating to insolvency, receivership, liquidation or composition for the benefit of creditors, if that petition or proceeding is not dismissed with prejudice within sixty (60) days after filing, or if you cease to do business; or (iii) if you commence or participate in any legal proceeding against NVIDIA, with respect to the Licensed Software that is the subject of the proceeding during the pendency of such legal proceeding. If you or your authorized NVIDIA reseller fail to pay license fees or service fees when due then NVIDIA may, in its sole discretion, suspend or terminate your license grants, services and any other rights provided under the AGREEMENT for the affected Licensed Software, in addition to any other remedies NVIDIA may have at law or equity. Upon any expiration or termination of the AGREEMENT, a license or a service provided hereunder, (a) any amounts owed to NVIDIA become immediately due and payable, (b) you must promptly discontinue use of the affected Licensed Software and/or service, and (c) you must promptly destroy or return to NVIDIA all copies of the affected Licensed Software and all portions thereof in your possession or control, and each party will promptly destroy or return to the other all of the other party’s Confidential Information within its possession or control. Upon written request, you will certify in writing that you have complied with your obligations under this section. Upon expiration or termination of the AGREEMENT all provisions survive except for the license grant provisions. + +9. CONSENT TO COLLECTION AND USE OF INFORMATION. + +You hereby agree and acknowledge that the Software may access and collect data about your Enterprise computer systems as well as configures the systems in order to (a) properly optimize such systems for use with the Software, (b) deliver content through the Software, (c) improve NVIDIA products and services, and (d) deliver marketing communications. Data collected by the Software includes, but is not limited to, system (i) hardware configuration and ID, (ii) operating system and driver configuration, (iii) installed applications, (iv) applications settings, performance, and usage data, and (iv) usage metrics of the Software. To the extent that you use the Software, you hereby consent to all of the foregoing, and represent and warrant that you have the right to grant such consent. In addition, you agree that you are solely responsible for maintaining appropriate data backups and system restore points for your Enterprise systems, and that NVIDIA will have no responsibility for any damage or loss to such systems (including loss of data or access) arising from or relating to (a) any changes to the configuration, application settings, environment variables, registry, drivers, BIOS, or other attributes of the systems (or any part of such systems) initiated through the Software; or (b) installation of any Software or third party software patches initiated through the Software. In certain systems you may change your system update preferences by unchecking "Automatically check for updates" in the "Preferences" tab of the control panel for the Software. + +In connection with the receipt of the Licensed Software or services you may receive access to links to third party websites and services and the availability of those links does not imply any endorsement by NVIDIA. NVIDIA encourages you to review the privacy statements on those sites and services that you choose to visit so that you can understand how they may collect, use and share personal information of individuals. NVIDIA is not responsible or liable for: (i) the availability or accuracy of such links; or (ii) the products, services or information available on or through such links; or (iii) the privacy statements or practices of sites and services controlled by other companies or organizations. + +To the extent that you or members of your Enterprise provide to NVIDIA during registration or otherwise personal data, you acknowledge that such information will be collected, used and disclosed by NVIDIA in accordance with NVIDIA's privacy policy, available at URL http://www.nvidia.com/object/privacy_policy.html. + +10. GENERAL. + +This SLA, any Supplements incorporated hereto, and Orders constitute the entire agreement of the parties with respect to the subject matter hereto and supersede all prior negotiations, conversations, or discussions between the parties relating to the subject matter hereto, oral or written, and all past dealings or industry custom. Any additional and/or conflicting terms and conditions on purchase order(s) or any other documents issued by you are null, void, and invalid. Any amendment or waiver under the AGREEMENT must be in writing and signed by representatives of both parties. + +The AGREEMENT and the rights and obligations thereunder may not be assigned by you, in whole or in part, including by merger, consolidation, dissolution, operation of law, or any other manner, without written consent of NVIDIA, and any purported assignment in violation of this provision shall be void and of no effect. NVIDIA may assign, delegate or transfer the AGREEMENT and its rights and obligations hereunder, and if to a non-Affiliate you will be notified. + +Each party acknowledges and agrees that the other is an independent contractor in the performance of the AGREEMENT, and each party is solely responsible for all of its employees, agents, contractors, and labor costs and expenses arising in connection therewith. The parties are not partners, joint ventures or otherwise affiliated, and neither has any authority to make any statements, representations or commitments of any kind to bind the other party without prior written consent. + +Neither party will be responsible for any failure or delay in its performance under the AGREEMENT (except for any payment obligations) to the extent due to causes beyond its reasonable control for so long as such force majeure event continues in effect. + +The AGREEMENT will be governed by and construed under the laws of the State of Delaware and the United States without regard to the conflicts of law provisions thereof and without regard to the United Nations Convention on Contracts for the International Sale of Goods. The parties consent to the personal jurisdiction of the federal and state courts located in Santa Clara County, California. You acknowledge and agree that a breach of any of your promises or agreements contained in the AGREEMENT may result in irreparable and continuing injury to NVIDIA for which monetary damages may not be an adequate remedy and therefore NVIDIA is entitled to seek injunctive relief as well as such other and further relief as may be appropriate. If any court of competent jurisdiction determines that any provision of the AGREEMENT is illegal, invalid or unenforceable, the remaining provisions will remain in full force and effect. Unless otherwise specified, remedies are cumulative. + +The Licensed Software has been developed entirely at private expense and is “commercial items” consisting of “commercial computer software” and “commercial computer software documentation” provided with RESTRICTED RIGHTS. Use, duplication or disclosure by the U.S. Government or a U.S. Government subcontractor is subject to the restrictions set forth in the AGREEMENT pursuant to DFARS 227.7202-3(a) or as set forth in subparagraphs (c)(1) and (2) of the Commercial Computer Software - Restricted Rights clause at FAR 52.227-19, as applicable. Contractor/manufacturer is NVIDIA, 2788 San Tomas Expressway, Santa Clara, CA 95051. + +You acknowledge that the Licensed Software described under the AGREEMENT is subject to export control under the U.S. Export Administration Regulations (EAR) and economic sanctions regulations administered by the U.S. Department of Treasury’s Office of Foreign Assets Control (OFAC). Therefore, you may not export, reexport or transfer in-country the Licensed Software without first obtaining any license or other approval that may be required by BIS and/or OFAC. You are responsible for any violation of the U.S. or other applicable export control or economic sanctions laws, regulations and requirements related to the Licensed Software. By accepting this SLA, you confirm that you are not a resident or citizen of any country currently embargoed by the U.S. and that you are not otherwise prohibited from receiving the Licensed Software. + +Any notice delivered by NVIDIA to you under the AGREEMENT will be delivered via mail, email or fax. Please direct your legal notices or other correspondence to NVIDIA Corporation, 2788 San Tomas Expressway, Santa Clara, California 95051, United States of America, Attention: Legal Department. + +GLOSSARY OF TERMS + +Certain capitalized terms, if not otherwise defined elsewhere in this SLA, shall have the meanings set forth below: + +a. “Affiliate” means any legal entity that Owns, is Owned by, or is commonly Owned with a party. “Own” means having more than 50% ownership or the right to direct the management of the entity. + +b. “AGREEMENT” means this SLA and all associated Supplements entered by the parties referencing this SLA. + +c. “Authorized Users” means your Enterprise individual employees and any of your Enterprise’s Contractors, subject to the terms of the “Enterprise and Contractors Usage” section. + +d. “Confidential Information” means the Licensed Software (unless made publicly available by NVIDIA without confidentiality obligations), and any NVIDIA business, marketing, pricing, research and development, know-how, technical, scientific, financial status, proposed new products or other information disclosed by NVIDIA to you which, at the time of disclosure, is designated in writing as confidential or proprietary (or like written designation), or orally identified as confidential or proprietary or is otherwise reasonably identifiable by parties exercising reasonable business judgment, as confidential. Confidential Information does not and will not include information that: (i) is or becomes generally known to the public through no fault of or breach of the AGREEMENT by the receiving party; (ii) is rightfully known by the receiving party at the time of disclosure without an obligation of confidentiality; (iii) is independently developed by the receiving party without use of the disclosing party’s Confidential Information; or (iv) is rightfully obtained by the receiving party from a third party without restriction on use or disclosure. + +e. “Contractor” means an individual who works primarily for your Enterprise on a contractor basis from your secure network. + +f. “Documentation” means the NVIDIA documentation made available for use with the Software, including (without limitation) user manuals, datasheets, operations instructions, installation guides, release notes and other materials provided to you under the AGREEMENT. + +g. “Enterprise” means you or any company or legal entity for which you accepted the terms of this SLA, and their subsidiaries of which your company or legal entity owns more than fifty percent (50%) of the issued and outstanding equity. + +h. “Excluded License” includes, without limitation, a software license that requires as a condition of use, modification, and/or distribution that software be (i) disclosed or distributed in source code form; (ii) licensed for the purpose of making derivative works; or (iii) redistributable at no charge. + +i. “Feedback” means any and all suggestions, feature requests, comments or other feedback regarding the Licensed Software, including possible enhancements or modifications thereto. + +j. “Intellectual Property Rights” means all patent, copyright, trademark, trade secret, trade dress, trade names, utility models, mask work, moral rights, rights of attribution or integrity service marks, master recording and music publishing rights, performance rights, author’s rights, database rights, registered design rights and any applications for the protection or registration of these rights, or other intellectual or industrial property rights or proprietary rights, howsoever arising and in whatever media, whether now known or hereafter devised, whether or not registered, (including all claims and causes of action for infringement, misappropriation or violation and all rights in any registrations and renewals), worldwide and whether existing now or in the future. + +k. “Licensed Software” means Software, Documentation and all modifications owned by NVIDIA. + +l. “Order” means a purchase order issued by you, a signed purchase agreement with you, or other ordering document issued by you to NVIDIA or a NVIDIA authorized reseller (including any on-line acceptance process) that references and incorporates the AGREEMENT and is accepted by NVIDIA. + +m. “Software” means the NVIDIA software programs licensed to you under the AGREEMENT including, without limitation, libraries, sample code, utility programs and programming code. + +n. “Supplement” means the additional terms and conditions beyond those stated in this SLA that apply to certain Licensed Software licensed hereunder. + +Notices +Notice +THE INFORMATION IN THIS GUIDE AND ALL OTHER INFORMATION CONTAINED IN NVIDIA DOCUMENTATION REFERENCED IN THIS GUIDE IS PROVIDED “AS IS.” NVIDIA MAKES NO WARRANTIES, EXPRESSED, IMPLIED, STATUTORY, OR OTHERWISE WITH RESPECT TO THE INFORMATION FOR THE PRODUCT, AND EXPRESSLY DISCLAIMS ALL IMPLIED WARRANTIES OF NONINFRINGEMENT, MERCHANTABILITY, AND FITNESS FOR A PARTICULAR PURPOSE. Notwithstanding any damages that customer might incur for any reason whatsoever, NVIDIA’s aggregate and cumulative liability towards customer for the product described in this guide shall be limited in accordance with the NVIDIA terms and conditions of sale for the product. + +THE NVIDIA PRODUCT DESCRIBED IN THIS GUIDE IS NOT FAULT TOLERANT AND IS NOT DESIGNED, MANUFACTURED OR INTENDED FOR USE IN CONNECTION WITH THE DESIGN, CONSTRUCTION, MAINTENANCE, AND/OR OPERATION OF ANY SYSTEM WHERE THE USE OR A FAILURE OF SUCH SYSTEM COULD RESULT IN A SITUATION THAT THREATENS THE SAFETY OF HUMAN LIFE OR SEVERE PHYSICAL HARM OR PROPERTY DAMAGE (INCLUDING, FOR EXAMPLE, USE IN CONNECTION WITH ANY NUCLEAR, AVIONICS, LIFE SUPPORT OR OTHER LIFE CRITICAL APPLICATION). NVIDIA EXPRESSLY DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY OF FITNESS FOR SUCH HIGH RISK USES. NVIDIA SHALL NOT BE LIABLE TO CUSTOMER OR ANY THIRD PARTY, IN WHOLE OR IN PART, FOR ANY CLAIMS OR DAMAGES ARISING FROM SUCH HIGH RISK USES. + +NVIDIA makes no representation or warranty that the product described in this guide will be suitable for any specified use without further testing or modification. Testing of all parameters of each product is not necessarily performed by NVIDIA. It is customer’s sole responsibility to ensure the product is suitable and fit for the application planned by customer and to do the necessary testing for the application in order to avoid a default of the application or the product. Weaknesses in customer’s product designs may affect the quality and reliability of the NVIDIA product and may result in additional or different conditions and/or requirements beyond those contained in this guide. NVIDIA does not accept any liability related to any default, damage, costs or problem which may be based on or attributable to: (i) the use of the NVIDIA product in any manner that is contrary to this guide, or (ii) customer product designs. + +Other than the right for customer to use the information in this guide with the product, no other license, either expressed or implied, is hereby granted by NVIDIA under this guide. Reproduction of information in this guide is permissible only if reproduction is approved by NVIDIA in writing, is reproduced without alteration, and is accompanied by all associated conditions, limitations, and notices. + +Trademarks +NVIDIA, the NVIDIA logo, and cuBLAS, CUDA, cuDNN, cuFFT, cuSPARSE, DIGITS, DGX, DGX-1, DGX Station, GRID, Jetson, Kepler, NGX, NVIDIA GPU Cloud, Maxwell, NCCL, NVLink, Pascal, Tegra, TensorRT, Tesla and Volta are trademarks and/or registered trademarks of NVIDIA Corporation in the Unites States and other countries. Other company and product names may be trademarks of the respective companies with which they are associated. + +Copyright +© 2021 NVIDIA Corporation. All rights reserved. \ No newline at end of file diff --git a/README.md b/README.md index 02447b6..c445e1a 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,123 @@ -# Isaac ROS Camera +# Isaac ROS Argus Camera +
+ +This repository provides monocular and stereo nodes that enable ROS developers to use cameras connected to Jetson platforms over a CSI interface. The nodes internally use libargus, which is an API for acquiring images and associated metadata from camera devices. + +[Libargus API reference](https://docs.nvidia.com/jetson/l4t-multimedia/group__LibargusAPI.html) + +This package is compatible with ROS2 Foxy and has been tested on the Jetson platfrom with off-the-shelf cameras from NVIDIA partners(see the **Reference Camera** section for more details). **Note**: x86_64 is not supported. + +## System Requirements +The CSI camera device needs to be connected and running and to present the video device node (e.g. /dev/video0). + +### Jetson +- [Jetson AGX Xavier and NX Xavier](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/) +- [JetPack 4.6](https://developer.nvidia.com/embedded/jetpack) + +**Note:** For best performance on Jetson, ensure that power settings are configured appropriately ([Power Management for Jetson](https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/power_management_jetson_xavier.html#wwpID0EUHA)). + +### Docker +Precompiled ROS2 Foxy packages are not available for JetPack 4.6 (based on Ubuntu 18.04 Bionic). You can either manually compile ROS2 Foxy and required dependent packages from source or use the Isaac ROS development Docker image from [Isaac ROS Common](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common). + +You must first install the [Nvidia Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html) to make use of the Docker container development/runtime environment. + +Configure `nvidia-container-runtime` as the default runtime for Docker by editing `/etc/docker/daemon.json` to include the following: +``` + "runtimes": { + "nvidia": { + "path": "nvidia-container-runtime", + "runtimeArgs": [] + } + }, + "default-runtime": "nvidia" +``` +and then restarting Docker: `sudo systemctl daemon-reload && sudo systemctl restart docker` + +Run the following script in `isaac_ros_common` to build the image and launch the container: + +`$ scripts/run_dev.sh ` + +You can either provide an optional path to mirror in your host ROS workspace with Isaac ROS packages, which will be made available in the container as `/workspaces/isaac_ros-dev`, or you can setup a new workspace in the container. + +### Reference Camera +The [Leopard Imaging](https://www.leopardimaging.com/product/) NVIDIA camera partner provides the below camera modules, which are compliant with the `isaac_ros_argus_camera` packages. + + +| Product Name | Type | Resolution | +| ------------ | ------------------- | ----------- | +| HAWK | Stereo Camera | 1920 x 1200 | +| OWL | Fisheye Camera | 1920 x 1200 | +| IMX477 | 4K Monocular Camera | 4056 x 3040 | + +## Quickstart +1. Create a ROS2 workspace if one is not already prepared: +`mkdir -p your_ws/src` +**Note:** The workspace can have any name; this guide assumes you name it `your_ws`. + +2. Clone the Isaac ROS Argus Camera repository to `your_ws/src/isaac_ros_argus_camera`. Check that you have [Git LFS](https://git-lfs.github.com/) installed before cloning to pull down all large files. +`sudo apt-get install git-lfs` +`cd your_ws/src && git clone https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_argus_camera` + +3. Build and source the workspace: +`cd your_ws && colcon build --symlink-install && source install/setup.bash` + +4. (Optional) Run tests to verify complete and correct installation: +`colcon test` + +5. Launch the node: +`ros2 run isaac_ros_argus_camera_mono isaac_ros_argus_camera_mono --ros-args -p device:=0 -p sensor:=0 -p output_encoding:="mono8"` + + +## Package Reference +### isaac_ros_argus_camera_mono + +`ros2 run isaac_ros_argus_camera_mono isaac_ros_argus_camera_mono --ros-args -p device:= -p sensor:= -p output_encoding:=` + +| ROS Argument | Usage | +| ----------------- | ---------------------------------------------------------------------- | +| `device_index` | The video node index (e.g. `/dev/video0`) | +| `sensor_index` | The sensor mode supported by the camera sensor and driver | +| `output_encoding` | The output image format. `mono8 ` and `rgb8 ` are currently supported | + +### isaac_ros_argus_camera_stereo + + `ros2 run isaac_ros_argus_camera_stereo isaac_ros_argus_camera_stereo --ros-args -p device:= -p sensor:= -p output_encoding:=` + +| ROS Argument | Usage | +| ----------------- | --------------------------------------------------------------------- | +| `device_index` | The first video node index (e.g. `/dev/video0`) | +| `sensor_index` | The sensor mode supported in the camera sensor and driver | +| `output_encoding` | The output image format. `mono8 ` and `rgb8 ` are currently supported | + +**Note**: To run the stereo node, two video nodes should present for left and right sensors, respectively (e.g. `/dev/video0` and `/dev/video1`). + +Examples: +`ros2 run isaac_ros_argus_camera_mono isaac_ros_argus_camera_mono --ros-args -p device:=0 -p sensor:=0 -p output_encoding:="rgb8"` + +`ros2 run isaac_ros_argus_camera_stereo isaac_ros_argus_camera_stereo --ros-args -p device:=0 -p sensor:=0 -p output_encoding:="mono8"` + +To view the output images: + + `ros2 run image_view image_saver --ros-args -r image:=/image_raw -p filename_format:="right_image.jpg" ` + + `ros2 run image_view image_saver --ros-args -r image:=/stereo/left/image_raw -p filename_format:="left_image.jpg" ` + +### CameraInfo Message +Argus nodes use the Argus Ext API to retrieve calibration parameters from the camera through the Linux device driver and convert it to [`CameraInfo`](http://docs.ros.org/en/melodic/api/sensor_msgs/html/msg/CameraInfo.html) messages. +Refer to [link](https://docs.nvidia.com/jetson/l4t-multimedia/classArgus_1_1Ext_1_1ISyncSensorCalibrationData.html) for the data structure of the calibration parameters. + +**Note**: Each camera module should have stored the calibration parameters in the internal memory like EEPROM and the device driver supports the API to extract it. Contact your camera vendor to get the driver that supports it. + +## Troubleshooting +### Argus camera nodes could stop publishing images sometimes when used with 4K high-resolution cameras in certain graph configurations on FastRTPS +With downstream subscribers that are not able to process images at the camera frame rate, we have observed instances where the Argus camera node suddenly stops publishing new images after about 15-20 minutes. + +#### Solution +We are continuing to collect data and diagnose the issue but changing the QoS settings to "Best Effort" and using a smaller frame size seem to alleviate the condition. + +# Updates + +| Date | Changes | +| -----| ------- | +| 2021-10-20 | Initial release | diff --git a/giistr-cla.md b/giistr-cla.md new file mode 100644 index 0000000..77e33bb --- /dev/null +++ b/giistr-cla.md @@ -0,0 +1,58 @@ +## Individual Contributor License Agreement (CLA) + +**Thank you for submitting your contributions to this project.** + +By signing this CLA, you agree that the following terms apply to all of your past, present and future contributions +to the project. + +### License. + +You hereby represent that all present, past and future contributions are governed by the +[MIT License](https://opensource.org/licenses/MIT) +copyright statement. + +This entails that to the extent possible under law, you transfer all copyright and related or neighboring rights +of the code or documents you contribute to the project itself or its maintainers. +Furthermore you also represent that you have the authority to perform the above waiver +with respect to the entirety of you contributions. + +### Moral Rights. + +To the fullest extent permitted under applicable law, you hereby waive, and agree not to +assert, all of your “moral rights” in or relating to your contributions for the benefit of the project. + +### Third Party Content. + +If your Contribution includes or is based on any source code, object code, bug fixes, configuration changes, tools, +specifications, documentation, data, materials, feedback, information or other works of authorship that were not +authored by you (“Third Party Content”) or if you are aware of any third party intellectual property or proprietary +rights associated with your Contribution (“Third Party Rights”), +then you agree to include with the submission of your Contribution full details respecting such Third Party +Content and Third Party Rights, including, without limitation, identification of which aspects of your +Contribution contain Third Party Content or are associated with Third Party Rights, the owner/author of the +Third Party Content and Third Party Rights, where you obtained the Third Party Content, and any applicable +third party license terms or restrictions respecting the Third Party Content and Third Party Rights. For greater +certainty, the foregoing obligations respecting the identification of Third Party Content and Third Party Rights +do not apply to any portion of a Project that is incorporated into your Contribution to that same Project. + +### Representations. + +You represent that, other than the Third Party Content and Third Party Rights identified by +you in accordance with this Agreement, you are the sole author of your Contributions and are legally entitled +to grant the foregoing licenses and waivers in respect of your Contributions. If your Contributions were +created in the course of your employment with your past or present employer(s), you represent that such +employer(s) has authorized you to make your Contributions on behalf of such employer(s) or such employer +(s) has waived all of their right, title or interest in or to your Contributions. + +### Disclaimer. + +To the fullest extent permitted under applicable law, your Contributions are provided on an "as is" +basis, without any warranties or conditions, express or implied, including, without limitation, any implied +warranties or conditions of non-infringement, merchantability or fitness for a particular purpose. You are not +required to provide support for your Contributions, except to the extent you desire to provide support. + +### No Obligation. + +You acknowledge that the maintainers of this project are under no obligation to use or incorporate your contributions +into the project. The decision to use or incorporate your contributions into the project will be made at the +sole discretion of the maintainers or their authorized delegates. \ No newline at end of file diff --git a/isaac_ros_argus_camera_mono/CMakeLists.txt b/isaac_ros_argus_camera_mono/CMakeLists.txt new file mode 100644 index 0000000..7ef5095 --- /dev/null +++ b/isaac_ros_argus_camera_mono/CMakeLists.txt @@ -0,0 +1,92 @@ +# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. +# +# NVIDIA CORPORATION and its licensors retain all intellectual property +# and proprietary rights in and to this software, related documentation +# and any modifications thereto. Any use, reproduction, disclosure or +# distribution of this software and related documentation without an express +# license agreement from NVIDIA CORPORATION is strictly prohibited. + +cmake_minimum_required(VERSION 3.5) +project(isaac_ros_argus_camera_mono) + +# Default to C++17 +if(NOT CMAKE_CXX_STANDARD) + set(CMAKE_CXX_STANDARD 17) +endif() + +if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang") + add_compile_options(-Wall -Wextra -Wpedantic) +endif() + +execute_process( + COMMAND uname -m + COMMAND tr -d '\n' + OUTPUT_VARIABLE ARCHITECTURE) +message(STATUS "Architecture: ${ARCHITECTURE}") + +find_package(rclcpp REQUIRED) +find_package(ament_cmake_auto REQUIRED) +ament_auto_find_build_dependencies() + +# Only build Argus nodes on Jetson platform +if(${ARCHITECTURE} STREQUAL "aarch64") + + # Find VPI dependency + find_package(vpi REQUIRED) + + # monocular camera node + ament_auto_add_library(monocular_node SHARED src/argus_camera_mono_node.cpp) + + target_include_directories( + monocular_node + PRIVATE /usr/lib/aarch64-linux-gnu/tegra + /usr/src/jetson_multimedia_api/argus/include + /usr/src/jetson_multimedia_api/argus/samples/utils + /usr/src/jetson_multimedia_api/include + include) + + target_compile_definitions(monocular_node PRIVATE "COMPOSITION_BUILDING_DLL") + + target_link_libraries( + monocular_node + /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so + /usr/lib/aarch64-linux-gnu/tegra/libnvargus_socketclient.so + /usr/lib/aarch64-linux-gnu/tegra/libnvargus_socketserver.so + /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so + vpi) + + target_compile_definitions(monocular_node PRIVATE "COMPOSITION_BUILDING_DLL") + + rclcpp_components_register_nodes(monocular_node "isaac_ros::argus::MonocularNode") + set(node_plugins "${node_plugins}isaac_ros::argus::MonocularNode;$\n") + + # isaac_ros_argus executable + ament_auto_add_executable(${PROJECT_NAME} src/argus_camera_mono_main.cpp) + + target_include_directories(${PROJECT_NAME} PUBLIC include) + + target_link_libraries(${PROJECT_NAME} monocular_node + ament_index_cpp::ament_index_cpp vpi) + + install( + TARGETS ${PROJECT_NAME} + ARCHIVE DESTINATION lib + LIBRARY DESTINATION lib + RUNTIME DESTINATION bin) + + install(DIRECTORY launch DESTINATION share/${PROJECT_NAME}) +endif() + +if(BUILD_TESTING) + find_package(ament_lint_auto REQUIRED) + + # Ignore copyright notices since we use custom JetPack EULA + set(ament_cmake_copyright_FOUND TRUE) + + ament_lint_auto_find_test_dependencies() + + find_package(launch_testing_ament_cmake REQUIRED) + +endif() + +ament_auto_package(INSTALL_TO_SHARE launch) diff --git a/isaac_ros_argus_camera_mono/include/isaac_ros_argus_camera_mono/argus_camera_mono_node.hpp b/isaac_ros_argus_camera_mono/include/isaac_ros_argus_camera_mono/argus_camera_mono_node.hpp new file mode 100644 index 0000000..148d44a --- /dev/null +++ b/isaac_ros_argus_camera_mono/include/isaac_ros_argus_camera_mono/argus_camera_mono_node.hpp @@ -0,0 +1,52 @@ +/** + * Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. + * + * NVIDIA CORPORATION and its licensors retain all intellectual property + * and proprietary rights in and to this software, related documentation + * and any modifications thereto. Any use, reproduction, disclosure or + * distribution of this software and related documentation without an express + * license agreement from NVIDIA CORPORATION is strictly prohibited. + */ + +#ifndef ISAAC_ROS_ARGUS_CAMERA_MONO__ARGUS_CAMERA_MONO_NODE_HPP_ +#define ISAAC_ROS_ARGUS_CAMERA_MONO__ARGUS_CAMERA_MONO_NODE_HPP_ + +#include + +#include "image_transport/image_transport.hpp" +#include "rclcpp/rclcpp.hpp" + +#define POLYNOMIAL_DISTORTION_COEFFICIENT_COUNT 8 +#define POLYNOMIAL_RADIAL_DISTORTION_COEFFICIENT_COUNT 6 + +namespace isaac_ros +{ +namespace argus +{ + +class MonocularNode : public rclcpp::Node +{ +public: + explicit MonocularNode(const rclcpp::NodeOptions &); + + virtual ~MonocularNode(); + +private: + // Publisher used for intra process comm + rclcpp::Publisher::SharedPtr image_pub_; + rclcpp::Publisher::SharedPtr camera_pub_; + + // Publisher used for inter process comm + image_transport::CameraPublisher publisher_; + + sensor_msgs::msg::CameraInfo::SharedPtr camerainfo_; + sensor_msgs::msg::Image::SharedPtr image_; + + uint32_t width_; + uint32_t height_; +}; + +} // namespace argus +} // namespace isaac_ros + +#endif // ISAAC_ROS_ARGUS_CAMERA_MONO__ARGUS_CAMERA_MONO_NODE_HPP_ diff --git a/isaac_ros_argus_camera_mono/launch/isaac_ros_argus_camera_mono_launch.py b/isaac_ros_argus_camera_mono/launch/isaac_ros_argus_camera_mono_launch.py new file mode 100644 index 0000000..fab6f10 --- /dev/null +++ b/isaac_ros_argus_camera_mono/launch/isaac_ros_argus_camera_mono_launch.py @@ -0,0 +1,34 @@ +# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. +# +# NVIDIA CORPORATION and its licensors retain all intellectual property +# and proprietary rights in and to this software, related documentation +# and any modifications thereto. Any use, reproduction, disclosure or +# distribution of this software and related documentation without an express +# license agreement from NVIDIA CORPORATION is strictly prohibited + +import launch +from launch_ros.actions import Node + + +def generate_launch_description(): + return launch.LaunchDescription([ + Node( + package='isaac_ros_argus_camera_mono', + executable='isaac_ros_argus_camera_mono', + parameters=[{ + 'sensor': 0, + 'device': 0, + 'output_encoding': 'rgb8' + }] + ), + Node( + package='image_view', + executable='image_saver', + remappings=[ + ('/image', '/image_raw') + ], + parameters=[{ + 'filename_format': 'image.jpg' + }] + ) + ]) diff --git a/isaac_ros_argus_camera_mono/launch/isaac_ros_argus_camera_mono_rectify_launch.py b/isaac_ros_argus_camera_mono/launch/isaac_ros_argus_camera_mono_rectify_launch.py new file mode 100644 index 0000000..2d77cf8 --- /dev/null +++ b/isaac_ros_argus_camera_mono/launch/isaac_ros_argus_camera_mono_rectify_launch.py @@ -0,0 +1,42 @@ +# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. +# +# NVIDIA CORPORATION and its licensors retain all intellectual property +# and proprietary rights in and to this software, related documentation +# and any modifications thereto. Any use, reproduction, disclosure or +# distribution of this software and related documentation without an express +# license agreement from NVIDIA CORPORATION is strictly prohibited. + +import launch +from launch_ros.actions import ComposableNodeContainer +from launch_ros.actions import Node +from launch_ros.descriptions import ComposableNode + + +def generate_launch_description(): + """Launch file which brings up argus monocular camera node with rectify node.""" + rectify_node = ComposableNode( + name='isaac_ros_rectify', + package='isaac_ros_image_proc', + plugin='isaac_ros::image_proc::RectifyNode', + remappings=[('image', 'image_color')]) + argus_camera_mono_node = Node( + package='isaac_ros_argus_camera_mono', + executable='isaac_ros_argus_camera_mono', + parameters=[{ + 'sensor': 0, + 'device': 0, + 'output_encoding': 'rgb8' + }], + remappings=[('image_raw', 'image_color')] + ) + argus_camera_mono_container = ComposableNodeContainer( + name='argus_camera_mono_container', + namespace='', + package='rclcpp_components', + executable='component_container', + composable_node_descriptions=[ + rectify_node + ], + output='screen' + ) + return launch.LaunchDescription([argus_camera_mono_container, argus_camera_mono_node]) diff --git a/isaac_ros_argus_camera_mono/package.xml b/isaac_ros_argus_camera_mono/package.xml new file mode 100644 index 0000000..3fe709e --- /dev/null +++ b/isaac_ros_argus_camera_mono/package.xml @@ -0,0 +1,40 @@ + + + + + + + isaac_ros_argus_camera_mono + 0.9.0 + Support for Argus Monocular Camera + + Greg Lo + JetPack EULA + https://developer.nvidia.com/blog/accelerating-ai-modules-for-ros-and-ros-2-on-jetson/ + Shashank Kumar + + ament_cmake_auto + + cv_bridge + image_transport + isaac_ros_common + rclcpp + rclcpp_components + + ament_lint_auto + ament_lint_common + isaac_ros_test + + + ament_cmake + + + diff --git a/isaac_ros_argus_camera_mono/src/argus_camera_mono_main.cpp b/isaac_ros_argus_camera_mono/src/argus_camera_mono_main.cpp new file mode 100644 index 0000000..1ea72d9 --- /dev/null +++ b/isaac_ros_argus_camera_mono/src/argus_camera_mono_main.cpp @@ -0,0 +1,36 @@ +/** + * Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. + * + * NVIDIA CORPORATION and its licensors retain all intellectual property + * and proprietary rights in and to this software, related documentation + * and any modifications thereto. Any use, reproduction, disclosure or + * distribution of this software and related documentation without an express + * license agreement from NVIDIA CORPORATION is strictly prohibited. + */ + +#include "isaac_ros_argus_camera_mono/argus_camera_mono_node.hpp" +#include +#include "rclcpp/rclcpp.hpp" + +int main(int argc, char * argv[]) +{ + rclcpp::init(argc, argv); + + rclcpp::executors::MultiThreadedExecutor exec; + + rclcpp::NodeOptions argus_monocular_options; + argus_monocular_options.arguments( + { + "--ros-args", + "-r", "__node:=argus_monocular" + }); + auto argus_monocular_node = std::make_shared( + argus_monocular_options); + exec.add_node(argus_monocular_node); + + // Spin with all the components loaded + exec.spin(); + + rclcpp::shutdown(); + return 0; +} diff --git a/isaac_ros_argus_camera_mono/src/argus_camera_mono_node.cpp b/isaac_ros_argus_camera_mono/src/argus_camera_mono_node.cpp new file mode 100644 index 0000000..06a34bd --- /dev/null +++ b/isaac_ros_argus_camera_mono/src/argus_camera_mono_node.cpp @@ -0,0 +1,603 @@ +/** + * Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. + * + * NVIDIA CORPORATION and its licensors retain all intellectual property + * and proprietary rights in and to this software, related documentation + * and any modifications thereto. Any use, reproduction, disclosure or + * distribution of this software and related documentation without an express + * license agreement from NVIDIA CORPORATION is strictly prohibited. + */ +#include "isaac_ros_argus_camera_mono/argus_camera_mono_node.hpp" + +// Ignore warnings in system library headers +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wpedantic" +#pragma GCC diagnostic ignored "-Wunused-parameter" +#pragma GCC diagnostic ignored "-Wmissing-field-initializers" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#pragma GCC diagnostic pop + +#include +#include +#include + +#include "cv_bridge/cv_bridge.h" +#include "isaac_ros_common/vpi_utilities.hpp" +#include "image_transport/image_transport.hpp" +#include "rclcpp/rclcpp.hpp" +#include "sensor_msgs/image_encodings.hpp" +#include "sensor_msgs/msg/camera_info.hpp" +#include "sensor_msgs/msg/image.hpp" + +// Ignore warnings in system library headers +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wpedantic" +#include "Thread.cpp" +#pragma GCC diagnostic pop + +namespace isaac_ros +{ +namespace argus +{ + +static const uint32_t NUM_BUFFERS = 10; + +// Argus Ext API to retrieve calibration parameters from driver +void getCalibrationData( + const Argus::Ext::ISyncSensorCalibrationData * sync_sensor_calibration_data, + sensor_msgs::msg::CameraInfo::SharedPtr camera_info) +{ + Argus::Size2D image_size = sync_sensor_calibration_data->getImageSizeInPixels(); + Argus::DistortionType lens_distortion_type = + sync_sensor_calibration_data->getLensDistortionType(); + + Argus::Point2D principal_point = sync_sensor_calibration_data->getPrincipalPoint(); + Argus::Point2D focal_length = sync_sensor_calibration_data->getFocalLength(); + + uint32_t radial_coeffs_count = sync_sensor_calibration_data->getRadialCoeffsCount( + lens_distortion_type); + std::vector dist; + sync_sensor_calibration_data->getRadialCoeffs(&dist, lens_distortion_type); + + float skew = sync_sensor_calibration_data->getSkew(); + + camera_info->width = image_size.width(); + camera_info->height = image_size.height(); + if (camera_info->width == 0 || camera_info->height == 0) { + RCLCPP_INFO( + rclcpp::get_logger( + "CameraCalibration"), + "Camera Calibration returned image dimension as 0. Is the camera properly calibrated?"); + return; + } + + // Filling in intrinsic camera matrix + camera_info->k[0] = (float64_t)focal_length.x(); + camera_info->k[1] = skew; + camera_info->k[2] = (float64_t)principal_point.x(); + camera_info->k[3] = 0; + camera_info->k[4] = (float64_t)focal_length.y(); + camera_info->k[5] = (float64_t)principal_point.y(); + camera_info->k[6] = 0; + camera_info->k[7] = 0; + camera_info->k[8] = 1; + + // For monocular camera, rotation matrix is identity matrix + camera_info->r[0] = 1; + camera_info->r[1] = 0; + camera_info->r[2] = 0; + camera_info->r[3] = 0; + camera_info->r[4] = 1; + camera_info->r[5] = 0; + camera_info->r[6] = 0; + camera_info->r[7] = 0; + camera_info->r[8] = 1; + + // Projection matrix specifying the camera intrinsics for rectified image + camera_info->p[0] = camera_info->k[0]; + camera_info->p[1] = camera_info->k[1]; + camera_info->p[2] = camera_info->k[2]; + camera_info->p[3] = 0; + camera_info->p[4] = camera_info->k[3]; + camera_info->p[5] = camera_info->k[4]; + camera_info->p[6] = camera_info->k[5]; + camera_info->p[7] = 0; + camera_info->p[8] = camera_info->k[6]; + camera_info->p[9] = camera_info->k[7]; + camera_info->p[10] = camera_info->k[8]; + camera_info->p[11] = 0; + + std::string distortion_type_polynomial = static_cast("DISTORTION_TYPE_POLYNOMIAL"); + std::string distortion_type_fisheye = static_cast("DISTORTION_TYPE_FISHEYE"); + + if (distortion_type_polynomial == lens_distortion_type.getName()) { + camera_info->distortion_model = static_cast("rational_polynomial"); + + uint32_t tangential_coeffs_count = sync_sensor_calibration_data->getTangentialCoeffsCount(); + std::vector tang; + sync_sensor_calibration_data->getTangentialCoeffs(&tang); + + // Filling in distortion parameters in the required format + if (radial_coeffs_count != 0 || tangential_coeffs_count != 0) { + camera_info->d.resize(radial_coeffs_count + tangential_coeffs_count); + camera_info->d[0] = (float64_t)dist[0]; + camera_info->d[1] = (float64_t)dist[1]; + for (uint32_t i = 2; i < 2 + tangential_coeffs_count; i++) { + camera_info->d[i] = (float64_t)tang[i - 2]; + } + for (uint32_t i = 2; i < radial_coeffs_count; i++) { + camera_info->d[i + tangential_coeffs_count] = (float64_t)dist[i]; + } + } + } else if (distortion_type_fisheye == lens_distortion_type.getName()) { + camera_info->distortion_model = static_cast("plumb_bob"); + if (radial_coeffs_count != 0) { + camera_info->d.resize(radial_coeffs_count); + for (uint32_t i = 0; i < radial_coeffs_count; i++) { + camera_info->d[i] = (float64_t)dist[i]; + } + } + } else { + RCLCPP_INFO( + rclcpp::get_logger( + "CameraCalibration"), + "Distortion Model is not supported by ROS. Currently polynomial and fisheye are supported."); + return; + } +} + +MonocularNode::MonocularNode(const rclcpp::NodeOptions & options) +: rclcpp::Node{"isaac_ros_argus_camera_mono", options} +{ + // Consumer Thread Class + class MonocularConsumerThread : public ArgusSamples::Thread + { +public: + explicit MonocularConsumerThread(MonocularNode * node, Argus::OutputStream * stream) + : node_{node}, stream_{stream} + { + } + + ~MonocularConsumerThread() + { + } + +private: + // Thread methods + virtual bool threadInitialize() + { + buffer_stream = Argus::interface_cast(stream_); + if (!buffer_stream) { + RCLCPP_ERROR( + node_->get_logger(), + "Failed to create buffer stream interface in consumer thread\n"); + return false; + } + + return true; + } + + virtual bool threadExecute() + { + VPIStream stream; + CHECK_STATUS(vpiStreamCreate(0, &stream)); + + uint64_t frame_id = 0; + + RCLCPP_INFO(node_->get_logger(), "Consumer Running"); + while (rclcpp::ok()) { + Argus::Status status = Argus::STATUS_OK; + Argus::Buffer * buf_ptr; + buf_ptr = buffer_stream->acquireBuffer(Argus::TIMEOUT_INFINITE, &status); + + if (status == Argus::STATUS_END_OF_STREAM) { + return true; + } + + Argus::IEGLImageBuffer * egl_image_buffer = Argus::interface_cast( + buf_ptr); + if (!egl_image_buffer) { + RCLCPP_ERROR( + node_->get_logger(), + "Failed to create IEGLImageBuffer interface\n"); + return false; + } + + EGLImageKHR egl_image = egl_image_buffer->getEGLImage(); + + // Map the encoding desired string to the number of channels needed + const std::unordered_map g_str_to_channels({ + {"mono8", CV_8UC1}, + {"mono16", CV_16UC1}, + {"bgr8", CV_8UC3}, + {"rgb8", CV_8UC3}, + {"bgra8", CV_8UC4}, + {"rgba8", CV_8UC4}}); + + std::string encoding; + node_->get_parameter("output_encoding", encoding); + VPIImageFormat vpi_image_format; + if (encoding == static_cast("mono8")) { + vpi_image_format = VPI_IMAGE_FORMAT_U8; + } else if (encoding == static_cast("rgb8")) { + vpi_image_format = VPI_IMAGE_FORMAT_RGB8; + } + + VPIWrapEGLImageParams vpi_egl_params; + vpi_egl_params.colorSpec = VPI_COLOR_SPEC_BT601_ER; + VPIImage vpi_image_in, vpi_image_out; + + CHECK_STATUS( + vpiImageCreate( + node_->width_, node_->height_, + vpi_image_format, 0, &vpi_image_out)); + + CHECK_STATUS(vpiImageCreateEGLImageWrapper(egl_image, &vpi_egl_params, 0, &vpi_image_in)); + + VPIConvertImageFormatParams vpi_convert_params; + CHECK_STATUS(vpiInitConvertImageFormatParams(&vpi_convert_params)); + + vpi_convert_params.policy = VPI_CONVERSION_CAST; + + CHECK_STATUS( + vpiSubmitConvertImageFormat( + stream, VPI_BACKEND_CUDA, vpi_image_in, vpi_image_out, + &vpi_convert_params)); + + CHECK_STATUS(vpiStreamSync(stream)); + + VPIImageData out_data; + vpiImageLock(vpi_image_out, VPI_LOCK_READ, &out_data); + cv::Mat out_mat{out_data.planes[0].height, out_data.planes[0].width, + g_str_to_channels.at(encoding), out_data.planes[0].data, + static_cast(out_data.planes[0].pitchBytes)}; + + buffer_stream->releaseBuffer(buf_ptr); + + cv_bridge::CvImage cv_img; + cv_img.image = out_mat; + cv_img.encoding = encoding; + cv_img.toImageMsg(*(node_->image_)); + + auto stamp = node_->now(); + node_->camerainfo_->header.stamp = node_->image_->header.stamp = stamp; + node_->camerainfo_->header.frame_id = node_->image_->header.frame_id = std::to_string( + frame_id++); + + if (node_->get_node_options().use_intra_process_comms()) { + RCLCPP_DEBUG_STREAM( + node_->get_logger(), + "Image message address [PUBLISH]:\t" << node_->image_.get()); + node_->image_pub_->publish(*(node_->image_)); + node_->camera_pub_->publish(*(node_->camerainfo_)); + } else { + node_->publisher_.publish(*(node_->image_), *(node_->camerainfo_)); + } + + vpiImageDestroy(vpi_image_in); + CHECK_STATUS(vpiImageUnlock(vpi_image_out)); + vpiImageDestroy(vpi_image_out); + + out_mat.release(); + } + + vpiStreamDestroy(stream); + RCLCPP_INFO(node_->get_logger(), "Consumer shutting down."); + requestShutdown(); + return true; + } + + virtual bool threadShutdown() {return true;} + + MonocularNode * node_; + Argus::OutputStream * stream_; + Argus::IBufferOutputStream * buffer_stream; + }; + + if (options.use_intra_process_comms()) { + image_pub_ = create_publisher("/image_raw", 20); + camera_pub_ = create_publisher("/camera_info", 20); + } else { + publisher_ = image_transport::create_camera_publisher(this, "/image_raw"); + } + + auto device_descriptor = rcl_interfaces::msg::ParameterDescriptor{}; + device_descriptor.description = "Index specifying camera device model"; + device_descriptor.read_only = false; + this->declare_parameter("device", 0, device_descriptor); + + auto sensor_descriptor = rcl_interfaces::msg::ParameterDescriptor{}; + sensor_descriptor.description = "Index specifying sensor mode"; + sensor_descriptor.read_only = false; + this->declare_parameter("sensor", 0, sensor_descriptor); + + auto output_encoding_description = rcl_interfaces::msg::ParameterDescriptor{}; + output_encoding_description.description = "ROS image encoding to use for the output image"; + output_encoding_description.additional_constraints = + "Currently supported: 'rgb8' or 'mono8' (Default: 'mono8')"; + this->declare_parameter("output_encoding", std::string{"mono8"}, output_encoding_description); + + // Set up Argus API Framework, identify available camera devices, and create a capture session for + // the selected device and sensor mode + uint32_t device_index, sensor_index, dvc_idx = 0, snsr_idx; + + Argus::UniqueObj cameraProvider(Argus::CameraProvider::create()); + Argus::ICameraProvider * camera_provider = Argus::interface_cast( + cameraProvider); + if (!camera_provider) { + RCLCPP_ERROR(this->get_logger(), "Cannot get core camera provider interface\n"); + return; + } + RCLCPP_INFO(this->get_logger(), "Argus Version: %s\n", camera_provider->getVersion().c_str()); + + // Get camera devices vector and display available camera device models + std::vector camera_devices; + Argus::Status status = camera_provider->getCameraDevices(&camera_devices); + if (camera_devices.empty()) { + RCLCPP_ERROR(this->get_logger(), "Failed to get camera devices"); + return; + } + + RCLCPP_INFO(this->get_logger(), "Set the camera model using the node parameter \"device\""); + RCLCPP_INFO(this->get_logger(), "Set the sensor mode using the node parameter \"sensor\""); + RCLCPP_INFO( + this->get_logger(), + "Set the output image format using the node parameter \"output_encoding\". " + "Supported encodings: \"mono8\" and \"rgb8\"\n"); + RCLCPP_INFO(this->get_logger(), "Following camera model indices are available:"); + + Argus::ICameraProperties * properties; + std::string model_string; + std::vector sensor_modes; + Argus::ISensorMode * sensor_properties; + Argus::Size2D res; + for (auto dvc : camera_devices) { + properties = Argus::interface_cast(dvc); + model_string = properties->getModuleString(); + RCLCPP_INFO(this->get_logger(), ""); + RCLCPP_INFO(this->get_logger(), "%d %s", dvc_idx++, model_string.c_str()); + RCLCPP_INFO(this->get_logger(), "Sensor modes supported for this camera device:"); + status = properties->getAllSensorModes(&sensor_modes); + if (status != Argus::STATUS_OK) { + RCLCPP_ERROR(this->get_logger(), "Failed to get sensor modes vector"); + return; + } + snsr_idx = 0; + for (auto snsr : sensor_modes) { + sensor_properties = Argus::interface_cast(snsr); + res = sensor_properties->getResolution(); + RCLCPP_INFO(this->get_logger(), "%d (%d x %d)", snsr_idx++, res.width(), res.height()); + } + } + + // Get the parameter value for camera device and get its properties + this->get_parameter("device", device_index); + Argus::CameraDevice * device = camera_devices[device_index]; + Argus::ICameraProperties * camera_properties = Argus::interface_cast( + device); + if (!camera_properties) { + RCLCPP_ERROR(this->get_logger(), "Failed to get ICameraProperties interface\n"); + return; + } + + // Get the parameter value for sensor model + this->get_parameter("sensor", sensor_index); + Argus::SensorMode * sensor_mode = sensor_modes[sensor_index]; + Argus::ISensorMode * i_sensor_mode = Argus::interface_cast(sensor_mode); + if (!i_sensor_mode) { + RCLCPP_ERROR(this->get_logger(), "Failed to get sensor mode interface"); + return; + } + RCLCPP_INFO(this->get_logger(), ""); + RCLCPP_INFO( + this->get_logger(), "Capturing from device %d using sensor mode %d (%dx%d)", + device_index, sensor_index, + i_sensor_mode->getResolution().width(), + i_sensor_mode->getResolution().height()); + + // Geting camera calibration data and putting it into sensor_msgs.camera_info + const Argus::Ext::ISyncSensorCalibrationData * sync_sensor_calibration_data = + Argus::interface_cast(device); + camerainfo_ = sensor_msgs::msg::CameraInfo::SharedPtr(new sensor_msgs::msg::CameraInfo()); + image_ = sensor_msgs::msg::Image::SharedPtr(new sensor_msgs::msg::Image()); + if (sync_sensor_calibration_data) { + getCalibrationData(sync_sensor_calibration_data, camerainfo_); + } else { + RCLCPP_INFO(this->get_logger(), "Cannot get ISyncSensorCalibrationData interface\n"); + } + + Argus::UniqueObj capture_session(camera_provider->createCaptureSession( + device, &status)); + + if (status != Argus::STATUS_OK) { + RCLCPP_ERROR(this->get_logger(), "Failed to create capture session\n"); + return; + } + + Argus::ICaptureSession * iSession = + Argus::interface_cast(capture_session); + if (!iSession) { + RCLCPP_ERROR(this->get_logger(), "Cannot get Capture Session interface\n"); + return; + } + + Argus::UniqueObj stream_settings( + iSession->createOutputStreamSettings(Argus::STREAM_TYPE_BUFFER)); + Argus::IBufferOutputStreamSettings * i_stream_settings = + Argus::interface_cast(stream_settings); + if (!i_stream_settings) { + RCLCPP_ERROR(this->get_logger(), "Cannot get IBufferOutputStreamSettings interface\n"); + return; + } + + i_stream_settings->setBufferType(Argus::BUFFER_TYPE_EGL_IMAGE); + if (i_stream_settings->setSyncType(Argus::SYNC_TYPE_NONE) != Argus::STATUS_OK) { + RCLCPP_ERROR(this->get_logger(), "Failed to disable EGL sync"); + return; + } + + Argus::UniqueObj output_stream(iSession->createOutputStream( + stream_settings.get())); + if (!output_stream) { + RCLCPP_ERROR(this->get_logger(), "Failed to create OutputStream\n"); + return; + } + Argus::IBufferOutputStream * i_buffer_output_stream = + Argus::interface_cast(output_stream); + if (!i_buffer_output_stream) { + RCLCPP_ERROR(this->get_logger(), "Failed to create IBufferOutputStream interface\n"); + return; + } + + Argus::Size2D size = + Argus::Size2D( + (i_sensor_mode->getResolution()).width(), + (i_sensor_mode->getResolution()).height() ); + + EGLint major, minor; + EGLDisplay egl_display = eglGetDisplay(EGL_DEFAULT_DISPLAY); + eglInitialize(egl_display, &major, &minor); + + Argus::UniqueObj buffer_settings( + i_buffer_output_stream->createBufferSettings()); + if (!buffer_settings) { + RCLCPP_ERROR(this->get_logger(), "Failed to create buffer settings\n"); + return; + } + + Argus::IEGLImageBufferSettings * i_buffer_settings = + Argus::interface_cast(buffer_settings); + if (!i_buffer_settings) { + RCLCPP_ERROR(this->get_logger(), "Cannot get IEGLImageBufferSettings interface\n"); + return; + } + + ArgusSamples::NativeBuffer * native_buffers[NUM_BUFFERS]; + EGLImageKHR egl_images[NUM_BUFFERS]; + Argus::UniqueObj buffers[NUM_BUFFERS]; + + ArgusSamples::NvNativeBuffer * buf; + + i_buffer_settings->setEGLDisplay(egl_display); + + // cppcheck-suppress syntaxError + for (uint32_t j = 0; j < NUM_BUFFERS; j++) { + buf = ArgusSamples::NvNativeBuffer::create(size, NvBufferColorFormat_NV12); + native_buffers[j] = (ArgusSamples::NativeBuffer *)buf; + if (!native_buffers[j]) { + RCLCPP_ERROR(this->get_logger(), "Failed to create native buffer\n"); + return; + } + + egl_images[j] = native_buffers[j]->createEGLImage(egl_display); + if (egl_images[j] == EGL_NO_IMAGE_KHR) { + RCLCPP_ERROR(this->get_logger(), "Failed to create eglImage from native buffer\n"); + return; + } + + i_buffer_settings->setEGLImage(egl_images[j]); + + buffers[j].reset(i_buffer_output_stream->createBuffer(buffer_settings.get())); + + if (!Argus::interface_cast(buffers[j])) { + RCLCPP_ERROR(this->get_logger(), "Failed to create buffer\n"); + return; + } + if (i_buffer_output_stream->releaseBuffer(buffers[j].get()) != Argus::STATUS_OK) { + RCLCPP_ERROR(this->get_logger(), "Failed to release buffer for capture use\n"); + return; + } + } + + std::string encoding; + this->get_parameter("output_encoding", encoding); + + Argus::UniqueObj request(iSession->createRequest()); + Argus::IRequest * i_request = Argus::interface_cast(request); + if (!i_request) { + RCLCPP_ERROR(this->get_logger(), "Failed to get capture request interface\n"); + return; + } + + status = i_request->enableOutputStream(output_stream.get()); + if (status != Argus::STATUS_OK) { + RCLCPP_ERROR(this->get_logger(), "Failed to enable stream in capture request\n"); + return; + } + + Argus::ISourceSettings * source_settings = + Argus::interface_cast(request); + if (!source_settings) { + RCLCPP_ERROR( + this->get_logger(), + "Failed to get source settings request interface\n"); + return; + } + + source_settings->setSensorMode(sensor_mode); + width_ = (i_sensor_mode->getResolution()).width(); + height_ = (i_sensor_mode->getResolution()).height(); + MonocularConsumerThread * mono_consumer_thread = new MonocularConsumerThread( + this, + output_stream.get()); + + if (!(mono_consumer_thread->initialize())) { + RCLCPP_ERROR(this->get_logger(), "Failed to initialize Consumer Thread\n"); + return; + } + if (!(mono_consumer_thread->waitRunning())) { + RCLCPP_ERROR(this->get_logger(), "Consumer Thread does not enter running state\n"); + return; + } + + RCLCPP_INFO(this->get_logger(), "Starting repeat capture requests."); + if (iSession->repeat(request.get()) != Argus::STATUS_OK) { + RCLCPP_ERROR(this->get_logger(), "Failed to start repeat capture request for preview"); + return; + } + + while (rclcpp::ok()) { + } + + iSession->stopRepeat(); + iSession->waitForIdle(); + + for (uint32_t j = 0; j < NUM_BUFFERS; j++) { + delete native_buffers[j]; + buffers[j].reset(); + } + RCLCPP_INFO(this->get_logger(), "Argus shutting down"); + buffer_settings.reset(); + output_stream.reset(); + stream_settings.reset(); + capture_session.reset(); + cameraProvider.reset(); +} + +MonocularNode::~MonocularNode() +{ +} + +} // namespace argus +} // namespace isaac_ros + +// Register as a component +#include "rclcpp_components/register_node_macro.hpp" +RCLCPP_COMPONENTS_REGISTER_NODE(isaac_ros::argus::MonocularNode) diff --git a/isaac_ros_argus_camera_stereo/CMakeLists.txt b/isaac_ros_argus_camera_stereo/CMakeLists.txt new file mode 100644 index 0000000..4ef518e --- /dev/null +++ b/isaac_ros_argus_camera_stereo/CMakeLists.txt @@ -0,0 +1,91 @@ +# Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. +# +# NVIDIA CORPORATION and its licensors retain all intellectual property +# and proprietary rights in and to this software, related documentation +# and any modifications thereto. Any use, reproduction, disclosure or +# distribution of this software and related documentation without an express +# license agreement from NVIDIA CORPORATION is strictly prohibited. + +cmake_minimum_required(VERSION 3.5) +project(isaac_ros_argus_camera_stereo) + +# Default to C++17 +if(NOT CMAKE_CXX_STANDARD) + set(CMAKE_CXX_STANDARD 17) +endif() + +if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang") + add_compile_options(-Wall -Wextra -Wpedantic) +endif() + +execute_process( + COMMAND uname -m + COMMAND tr -d '\n' + OUTPUT_VARIABLE ARCHITECTURE) +message(STATUS "Architecture: ${ARCHITECTURE}") + +find_package(rclcpp REQUIRED) +find_package(ament_cmake_auto REQUIRED) +ament_auto_find_build_dependencies() + +# Only build Argus nodes on Jetson platform +if(${ARCHITECTURE} STREQUAL "aarch64") + # Find VPI dependency + find_package(vpi REQUIRED) + + # stereo camera node + ament_auto_add_library(stereo_node SHARED src/argus_camera_stereo_node.cpp) + + target_include_directories( + stereo_node + PRIVATE /usr/lib/aarch64-linux-gnu/tegra + /usr/src/jetson_multimedia_api/argus/include + /usr/src/jetson_multimedia_api/argus/samples/utils + /usr/src/jetson_multimedia_api/include + include) + + target_compile_definitions(stereo_node PRIVATE "COMPOSITION_BUILDING_DLL") + + target_link_libraries( + stereo_node + /usr/lib/aarch64-linux-gnu/tegra/libnvargus.so + /usr/lib/aarch64-linux-gnu/tegra/libnvargus_socketclient.so + /usr/lib/aarch64-linux-gnu/tegra/libnvargus_socketserver.so + /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so + vpi) + + target_compile_definitions(stereo_node PRIVATE "COMPOSITION_BUILDING_DLL") + + rclcpp_components_register_nodes(stereo_node "isaac_ros::argus::StereoNode") + set(node_plugins + "${node_plugins}isaac_ros::argus::StereoNode;$\n" + ) + + # isaac_ros_argus executable + ament_auto_add_executable(${PROJECT_NAME} src/argus_camera_stereo_main.cpp) + + target_include_directories(${PROJECT_NAME} PUBLIC include) + + target_link_libraries(${PROJECT_NAME} stereo_node + ament_index_cpp::ament_index_cpp vpi) + + install( + TARGETS ${PROJECT_NAME} + ARCHIVE DESTINATION lib + LIBRARY DESTINATION lib + RUNTIME DESTINATION bin) +endif() + +if(BUILD_TESTING) + find_package(ament_lint_auto REQUIRED) + + # Ignore copyright notices since we use custom JetPack EULA + set(ament_cmake_copyright_FOUND TRUE) + + ament_lint_auto_find_test_dependencies() + + find_package(launch_testing_ament_cmake REQUIRED) + +endif() + +ament_auto_package() diff --git a/isaac_ros_argus_camera_stereo/include/isaac_ros_argus_camera_stereo/argus_camera_stereo_node.hpp b/isaac_ros_argus_camera_stereo/include/isaac_ros_argus_camera_stereo/argus_camera_stereo_node.hpp new file mode 100644 index 0000000..51dc52e --- /dev/null +++ b/isaac_ros_argus_camera_stereo/include/isaac_ros_argus_camera_stereo/argus_camera_stereo_node.hpp @@ -0,0 +1,53 @@ +/** + * Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. + * + * NVIDIA CORPORATION and its licensors retain all intellectual property + * and proprietary rights in and to this software, related documentation + * and any modifications thereto. Any use, reproduction, disclosure or + * distribution of this software and related documentation without an express + * license agreement from NVIDIA CORPORATION is strictly prohibited. + */ + +#ifndef ISAAC_ROS_ARGUS_CAMERA_STEREO__ARGUS_CAMERA_STEREO_NODE_HPP_ +#define ISAAC_ROS_ARGUS_CAMERA_STEREO__ARGUS_CAMERA_STEREO_NODE_HPP_ + +#include +#include + +#include "image_transport/image_transport.hpp" +#include "rclcpp/rclcpp.hpp" + +#define POLYNOMIAL_DISTORTION_COEFFICIENT_COUNT 8 +#define POLYNOMIAL_RADIAL_DISTORTION_COEFFICIENT_COUNT 6 + +#define MAX_MODULE_STRING 32 +#define MAX_CAM_DEVICE 6 + +namespace isaac_ros +{ +namespace argus +{ + +class ArgusStereoNode : public rclcpp::Node +{ +public: + explicit ArgusStereoNode(const rclcpp::NodeOptions &); + + virtual ~ArgusStereoNode(); + +private: + // Publishers used for inter process comm + image_transport::CameraPublisher publishers_[MAX_CAM_DEVICE]; + + sensor_msgs::msg::CameraInfo::SharedPtr camerainfo_[MAX_CAM_DEVICE]; + sensor_msgs::msg::Image::SharedPtr images_[MAX_CAM_DEVICE]; + + std::string encoding_; + uint32_t width_; + uint32_t height_; +}; + +} // namespace argus +} // namespace isaac_ros + +#endif // ISAAC_ROS_ARGUS_CAMERA_STEREO__ARGUS_CAMERA_STEREO_NODE_HPP_ diff --git a/isaac_ros_argus_camera_stereo/package.xml b/isaac_ros_argus_camera_stereo/package.xml new file mode 100644 index 0000000..00ef916 --- /dev/null +++ b/isaac_ros_argus_camera_stereo/package.xml @@ -0,0 +1,39 @@ + + + + + + + isaac_ros_argus_camera_stereo + 0.9.0 + Support for Argus Stereo Camera + + Greg Lo + JetPack EULA + https://developer.nvidia.com/blog/accelerating-ai-modules-for-ros-and-ros-2-on-jetson/ + Shashank Kumar + + ament_cmake_auto + + cv_bridge + image_transport + isaac_ros_common + rclcpp + rclcpp_components + + ament_lint_auto + ament_lint_common + isaac_ros_test + + + ament_cmake + + diff --git a/isaac_ros_argus_camera_stereo/src/argus_camera_stereo_main.cpp b/isaac_ros_argus_camera_stereo/src/argus_camera_stereo_main.cpp new file mode 100644 index 0000000..08f973b --- /dev/null +++ b/isaac_ros_argus_camera_stereo/src/argus_camera_stereo_main.cpp @@ -0,0 +1,36 @@ +/** + * Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. + * + * NVIDIA CORPORATION and its licensors retain all intellectual property + * and proprietary rights in and to this software, related documentation + * and any modifications thereto. Any use, reproduction, disclosure or + * distribution of this software and related documentation without an express + * license agreement from NVIDIA CORPORATION is strictly prohibited. + */ + +#include "isaac_ros_argus_camera_stereo/argus_camera_stereo_node.hpp" + +#include + +#include "rclcpp/rclcpp.hpp" + +int main(int argc, char * argv[]) +{ + rclcpp::init(argc, argv); + + rclcpp::executors::MultiThreadedExecutor exec; + + rclcpp::NodeOptions argus_stereo_options; + argus_stereo_options.arguments( + {"--ros-args", + "-r", "__node:=argus_stereo"}); + auto argus_stereo_node = + std::make_shared(argus_stereo_options); + exec.add_node(argus_stereo_node); + + // Spin with all the components loaded + exec.spin(); + + rclcpp::shutdown(); + return 0; +} diff --git a/isaac_ros_argus_camera_stereo/src/argus_camera_stereo_node.cpp b/isaac_ros_argus_camera_stereo/src/argus_camera_stereo_node.cpp new file mode 100644 index 0000000..679960d --- /dev/null +++ b/isaac_ros_argus_camera_stereo/src/argus_camera_stereo_node.cpp @@ -0,0 +1,900 @@ +/** + * Copyright (c) 2021, NVIDIA CORPORATION. All rights reserved. + * + * NVIDIA CORPORATION and its licensors retain all intellectual property + * and proprietary rights in and to this software, related documentation + * and any modifications thereto. Any use, reproduction, disclosure or + * distribution of this software and related documentation without an express + * license agreement from NVIDIA CORPORATION is strictly prohibited. + */ + +#include "isaac_ros_argus_camera_stereo/argus_camera_stereo_node.hpp" + +// Ignore warnings in system library headers +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wpedantic" +#pragma GCC diagnostic ignored "-Wunused-parameter" +#pragma GCC diagnostic ignored "-Wmissing-field-initializers" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#pragma GCC diagnostic pop + +#include +#include +#include + +#include "cv_bridge/cv_bridge.h" +#include "isaac_ros_common/vpi_utilities.hpp" +#include "image_transport/image_transport.hpp" +#include "rclcpp/rclcpp.hpp" +#include "rclcpp/logger.hpp" +#include "sensor_msgs/image_encodings.hpp" +#include "sensor_msgs/msg/camera_info.hpp" +#include "sensor_msgs/msg/image.hpp" + +// Ignore warnings in system library headers +#pragma GCC diagnostic push +#pragma GCC diagnostic ignored "-Wpedantic" +#include "Thread.cpp" +#pragma GCC diagnostic pop + +namespace isaac_ros +{ +namespace argus +{ + +static const uint32_t NUM_BUFFERS = 10; + +void SyncStereoCalibrationData( + const Argus::Ext::ISyncSensorCalibrationData * sync_sensor_calibration_data, + sensor_msgs::msg::CameraInfo::SharedPtr camera_info) +{ + Argus::Size2D image_size = sync_sensor_calibration_data->getImageSizeInPixels(); + Argus::DistortionType lens_distortion_type = + sync_sensor_calibration_data->getLensDistortionType(); + + Argus::Point2D principal_point = sync_sensor_calibration_data->getPrincipalPoint(); + Argus::Point2D focal_length = sync_sensor_calibration_data->getFocalLength(); + + uint32_t radial_coeffs_count = sync_sensor_calibration_data->getRadialCoeffsCount( + lens_distortion_type); + std::vector dist; + sync_sensor_calibration_data->getRadialCoeffs(&dist, lens_distortion_type); + + Argus::Point3D rot3d = sync_sensor_calibration_data->getRotationParams(); + Argus::Point3D translation = sync_sensor_calibration_data->getTranslationParams(); + + float skew = sync_sensor_calibration_data->getSkew(); + + camera_info->width = image_size.width(); + camera_info->height = image_size.height(); + camera_info->distortion_model = static_cast(lens_distortion_type.getName()); + + if (camera_info->width == 0 || camera_info->height == 0) { + RCLCPP_INFO( + rclcpp::get_logger( + "CameraCalibration"), + "Camera Calibration returned image dimension as 0. Is the camera properly calibrated?"); + return; + } + + // Filling in intrinsic camera matrix + camera_info->k[0] = (float64_t)focal_length.x(); + camera_info->k[1] = skew; + camera_info->k[2] = (float64_t)principal_point.x(); + camera_info->k[3] = 0; + camera_info->k[4] = (float64_t)focal_length.y(); + camera_info->k[5] = (float64_t)principal_point.y(); + camera_info->k[6] = 0; + camera_info->k[7] = 0; + camera_info->k[8] = 1; + + // Compute rotation matrix + float64_t rx = (float64_t)rot3d.x(); + float64_t ry = (float64_t)rot3d.y(); + float64_t rz = (float64_t)rot3d.z(); + + if (rx != 0 || ry != 0 || rz != 0) { + float64_t angle = sqrt(rx * rx + ry * ry + rz * rz); + std::vector axis{rx / angle, ry / angle, rz / angle}; + float64_t s = sin(angle), c = cos(angle); + float64_t * rm = new float64_t[9]; + + // Create cross product matrix for axis vector + rm[0] = 0; + rm[1] = -axis[2]; + rm[2] = axis[1]; + rm[3] = axis[2]; + rm[4] = 0; + rm[5] = -axis[0]; + rm[6] = -axis[1]; + rm[7] = axis[0]; + rm[8] = 0; + + // Fill Rotation matrix + camera_info->r[0] = c + (1 - c) * axis[0] * axis[0] + s * rm[0]; + camera_info->r[1] = (1 - c) * axis[0] * axis[1] + s * rm[1]; + camera_info->r[2] = (1 - c) * axis[0] * axis[2] + s * rm[2]; + camera_info->r[3] = (1 - c) * axis[1] * axis[0] + s * rm[3]; + camera_info->r[4] = c + (1 - c) * axis[1] * axis[1] + s * rm[4]; + camera_info->r[5] = (1 - c) * axis[1] * axis[2] + s * rm[5]; + camera_info->r[6] = (1 - c) * axis[2] * axis[0] + s * rm[6]; + camera_info->r[7] = (1 - c) * axis[2] * axis[1] + s * rm[7]; + camera_info->r[8] = c + (1 - c) * axis[2] * axis[2] + s * rm[8]; + + delete[] rm; + } + + // Projection matrix specifying the camera intrinsics for rectified image + camera_info->p[0] = camera_info->k[0]; + camera_info->p[1] = camera_info->k[1]; + camera_info->p[2] = camera_info->k[2]; + camera_info->p[3] = translation.x(); + camera_info->p[4] = camera_info->k[3]; + camera_info->p[5] = camera_info->k[4]; + camera_info->p[6] = camera_info->k[5]; + camera_info->p[7] = translation.y(); + camera_info->p[8] = camera_info->k[6]; + camera_info->p[9] = camera_info->k[7]; + camera_info->p[10] = camera_info->k[8]; + camera_info->p[11] = translation.z(); + + std::string distortion_type_polynomial = static_cast("DISTORTION_TYPE_POLYNOMIAL"); + std::string distortion_type_fisheye = static_cast("DISTORTION_TYPE_FISHEYE"); + + if (distortion_type_polynomial == lens_distortion_type.getName()) { + camera_info->distortion_model = static_cast("rational_polynomial"); + + uint32_t tangential_coeffs_count = sync_sensor_calibration_data->getTangentialCoeffsCount(); + std::vector tang; + sync_sensor_calibration_data->getTangentialCoeffs(&tang); + + // Filling in distortion parameters in the required format + if (radial_coeffs_count != 0 || tangential_coeffs_count != 0) { + camera_info->d.resize(radial_coeffs_count + tangential_coeffs_count); + camera_info->d[0] = (float64_t)dist[0]; + camera_info->d[1] = (float64_t)dist[1]; + for (uint32_t i = 2; i < 2 + tangential_coeffs_count; i++) { + camera_info->d[i] = (float64_t)tang[i - 2]; + } + for (uint32_t i = 2; i < radial_coeffs_count; i++) { + camera_info->d[i + tangential_coeffs_count] = (float64_t)dist[i]; + } + } + } else if (distortion_type_fisheye == lens_distortion_type.getName()) { + camera_info->distortion_model = static_cast("plumb_bob"); + + if (radial_coeffs_count != 0) { + camera_info->d.resize(radial_coeffs_count); + for (uint32_t i = 0; i < radial_coeffs_count; i++) { + camera_info->d[i] = (float64_t)dist[i]; + } + } + } else { + RCLCPP_INFO( + rclcpp::get_logger( + "CameraCalibration"), + "Distortion Model is not supported by ROS. Currently polynomial and fisheye are supported."); + return; + } +} + +void ComputeStereoRectificationData(sensor_msgs::msg::CameraInfo::SharedPtr camerainfo[]) +{ + std::string distortion_type_polynomial = static_cast("DISTORTION_TYPE_POLYNOMIAL"); + std::string distortion_type_fisheye = static_cast("DISTORTION_TYPE_FISHEYE"); + + cv::Matx33d camera_intrinsics[2]; + cv::Matx polynomial_distortion_parameters[2]; + cv::Matx fisheye_distortion_parameters[2]; + cv::Matx31d translation_parameters; + cv::Matx33d rectification_matrix[2]; + cv::Matx34d projection_matrix[2]; + cv::Matx44d disparity_to_depth_mapping; + cv::Matx33d rotation_matrix; + + for (uint32_t i = 0; i < 2; i++) { + camera_intrinsics[i] = { + camerainfo[i]->k[0], + camerainfo[i]->k[1], + camerainfo[i]->k[2], + camerainfo[i]->k[3], + camerainfo[i]->k[4], + camerainfo[i]->k[5], + camerainfo[i]->k[6], + camerainfo[i]->k[7], + camerainfo[i]->k[8] + }; + + if (camerainfo[i]->distortion_model == distortion_type_polynomial) { + polynomial_distortion_parameters[i] = { + camerainfo[i]->d[0], + camerainfo[i]->d[1], + camerainfo[i]->d[6], + camerainfo[i]->d[7], + camerainfo[i]->d[2], + camerainfo[i]->d[3], + camerainfo[i]->d[4], + camerainfo[i]->d[5] + }; + } else { + fisheye_distortion_parameters[i] = { + camerainfo[i]->d[0], + camerainfo[i]->d[1], + camerainfo[i]->d[2], + camerainfo[i]->d[3] + }; + } + } + + rotation_matrix = { + camerainfo[0]->r[0], + camerainfo[0]->r[1], + camerainfo[0]->r[2], + camerainfo[0]->r[3], + camerainfo[0]->r[4], + camerainfo[0]->r[5], + camerainfo[0]->r[6], + camerainfo[0]->r[7], + camerainfo[0]->r[8] + }; + + translation_parameters = { + camerainfo[0]->p[3], + camerainfo[0]->p[7], + camerainfo[0]->p[11] + }; + + cv::stereoRectify( + camera_intrinsics[0], polynomial_distortion_parameters[0], + camera_intrinsics[1], polynomial_distortion_parameters[1], + cv::Size2i(camerainfo[0]->width, camerainfo[0]->height), + rotation_matrix, translation_parameters, rectification_matrix[0], rectification_matrix[1], + projection_matrix[0], projection_matrix[1], + disparity_to_depth_mapping, cv::CALIB_ZERO_DISPARITY); + + for (uint32_t i = 0; i < 2; i++) { + camerainfo[i]->r = { + rectification_matrix[i](0, 0), + rectification_matrix[i](0, 1), + rectification_matrix[i](0, 2), + rectification_matrix[i](1, 0), + rectification_matrix[i](1, 1), + rectification_matrix[i](1, 2), + rectification_matrix[i](2, 0), + rectification_matrix[i](2, 1), + rectification_matrix[i](2, 2) + }; + camerainfo[i]->p = { + projection_matrix[i](0, 0), + projection_matrix[i](0, 1), + projection_matrix[i](0, 2), + projection_matrix[i](0, 3), + projection_matrix[i](1, 0), + projection_matrix[i](1, 1), + projection_matrix[i](1, 2), + projection_matrix[i](1, 3), + projection_matrix[i](2, 0), + projection_matrix[i](2, 1), + projection_matrix[i](2, 2), + projection_matrix[i](2, 3) + }; + } +} + +ArgusStereoNode::ArgusStereoNode(const rclcpp::NodeOptions & options) +: rclcpp::Node{"argus_stereo", options} +{ + class SyncStereoConsumerThread; + + typedef struct + { + char moduleName[MAX_MODULE_STRING]; + uint32_t camDevice[MAX_CAM_DEVICE]; + Argus::UniqueObj stream[MAX_CAM_DEVICE]; + Argus::UniqueObj capture_session; + Argus::UniqueObj stream_settings; + SyncStereoConsumerThread * sync_stereo_consumer; + uint32_t sensor_count; + bool initialized; + } ModuleInfo; + + class SyncStereoConsumerThread : public ArgusSamples::Thread + { +public: + explicit SyncStereoConsumerThread( + ModuleInfo * module_Info, ArgusStereoNode * node) + : node_{node} + { + left_stream = module_Info->stream[0].get(); + if (module_Info->sensor_count > 1) { + right_stream = module_Info->stream[1].get(); + } else { + right_stream = NULL; + } + } + + ~SyncStereoConsumerThread() + { + } + +private: + // Thread methods + + virtual bool threadInitialize() + { + left_buffer_stream = Argus::interface_cast(left_stream); + if (!left_buffer_stream) { + RCLCPP_ERROR( + node_->get_logger(), "Failed to create left buffer stream interface\n"); + return false; + } + + if (right_stream) { + right_buffer_stream = Argus::interface_cast(right_stream); + if (!right_buffer_stream) { + RCLCPP_ERROR( + node_->get_logger(), "Failed to create right buffer stream interface\n"); + return false; + } + } + + return true; + } + + virtual bool threadExecute() + { + Argus::Status status = Argus::STATUS_OK; + Argus::Buffer * left_buf, * right_buf; + EGLImageKHR left_image, right_image; + Argus::IEGLImageBuffer * egl_image_buffer; + + VPIStream stream; + CHECK_STATUS(vpiStreamCreate(0, &stream)); + + VPIConvertImageFormatParams vpi_convert_params; + CHECK_STATUS(vpiInitConvertImageFormatParams(&vpi_convert_params)); + + vpi_convert_params.policy = VPI_CONVERSION_CAST; + + // Map the encoding desired string to the number of channels needed + const std::unordered_map g_str_to_channels({{"mono8", CV_8UC1}, + {"mono16", CV_16UC1}, + {"bgr8", CV_8UC3}, + {"rgb8", CV_8UC3}, + {"bgra8", CV_8UC4}, + {"rgba8", CV_8UC4}}); + + VPIImageFormat vpi_image_format; + if (node_->encoding_ == static_cast("mono8")) { + vpi_image_format = VPI_IMAGE_FORMAT_U8; + } else if (node_->encoding_ == static_cast("rgb8")) { + vpi_image_format = VPI_IMAGE_FORMAT_RGB8; + } + + VPIWrapEGLImageParams vpi_egl_params; + vpi_egl_params.colorSpec = VPI_COLOR_SPEC_BT601_ER; + VPIImage vpi_image_left_in, vpi_image_left_out, vpi_image_right_in, vpi_image_right_out; + + uint64_t frame_id = 0; + + RCLCPP_INFO(node_->get_logger(), "Consumer Running"); + while (rclcpp::ok()) { + left_buf = left_buffer_stream->acquireBuffer(Argus::TIMEOUT_INFINITE, &status); + if (status == Argus::STATUS_END_OF_STREAM) { + break; + } + egl_image_buffer = Argus::interface_cast(left_buf); + left_image = egl_image_buffer->getEGLImage(); + status = Argus::STATUS_OK; + if (right_buffer_stream) { + right_buf = right_buffer_stream->acquireBuffer(Argus::TIMEOUT_INFINITE, &status); + if (status == Argus::STATUS_END_OF_STREAM) { + break; + } + egl_image_buffer = Argus::interface_cast(right_buf); + right_image = egl_image_buffer->getEGLImage(); + } + + CHECK_STATUS( + vpiImageCreate( + node_->width_, node_->height_, vpi_image_format, 0, + &vpi_image_left_out)); + + CHECK_STATUS( + vpiImageCreate( + node_->width_, node_->height_, vpi_image_format, 0, + &vpi_image_right_out)); + + CHECK_STATUS( + vpiImageCreateEGLImageWrapper( + left_image, &vpi_egl_params, 0, + &vpi_image_left_in)); + + CHECK_STATUS( + vpiImageCreateEGLImageWrapper( + right_image, &vpi_egl_params, 0, + &vpi_image_right_in)); + + CHECK_STATUS( + vpiSubmitConvertImageFormat( + stream, VPI_BACKEND_CUDA, vpi_image_left_in, + vpi_image_left_out, &vpi_convert_params)); + + CHECK_STATUS( + vpiSubmitConvertImageFormat( + stream, VPI_BACKEND_CUDA, vpi_image_right_in, + vpi_image_right_out, &vpi_convert_params)); + + CHECK_STATUS(vpiStreamSync(stream)); + + VPIImageData out_data_left, out_data_right; + + CHECK_STATUS(vpiImageLock(vpi_image_left_out, VPI_LOCK_READ, &out_data_left)); + CHECK_STATUS(vpiImageLock(vpi_image_right_out, VPI_LOCK_READ, &out_data_right)); + + cv_bridge::CvImage cv_img; + + auto stamp = node_->now(); + + cv_img.encoding = node_->encoding_; + + cv::Mat out_mat_left{out_data_left.planes[0].height, out_data_left.planes[0].width, + g_str_to_channels.at(node_->encoding_), out_data_left.planes[0].data, + static_cast(out_data_left.planes[0].pitchBytes)}; + cv_img.image = out_mat_left; + cv_img.toImageMsg(*(node_->images_[0])); + left_buffer_stream->releaseBuffer(left_buf); + + cv::Mat out_mat_right{out_data_right.planes[0].height, out_data_right.planes[0].width, + g_str_to_channels.at(node_->encoding_), out_data_right.planes[0].data, + static_cast(out_data_right.planes[0].pitchBytes)}; + cv_img.image = out_mat_right; + cv_img.toImageMsg(*(node_->images_[1])); + right_buffer_stream->releaseBuffer(right_buf); + + CHECK_STATUS(vpiImageUnlock(vpi_image_right_out)); + CHECK_STATUS(vpiImageUnlock(vpi_image_left_out)); + vpiImageDestroy(vpi_image_left_out); + vpiImageDestroy(vpi_image_right_out); + vpiImageDestroy(vpi_image_left_in); + vpiImageDestroy(vpi_image_right_in); + + node_->images_[0]->header.stamp = node_->images_[1]->header.stamp = + node_->camerainfo_[0]->header.stamp = + node_->camerainfo_[1]->header.stamp = stamp; + node_->images_[0]->header.frame_id = node_->images_[1]->header.frame_id = + node_->camerainfo_[0]->header.frame_id = node_->camerainfo_[1]->header.frame_id = + std::to_string(frame_id++); + + node_->publishers_[0].publish(*(node_->images_[0]), *(node_->camerainfo_[0])); + node_->publishers_[1].publish(*(node_->images_[1]), *(node_->camerainfo_[1])); + + out_mat_left.release(); + out_mat_right.release(); + } + + RCLCPP_INFO(node_->get_logger(), "Consumer shutting down."); + requestShutdown(); + return true; + } + + virtual bool threadShutdown() + { + return true; + } + + Argus::OutputStream * left_stream; + Argus::OutputStream * right_stream; + + Argus::IBufferOutputStream * left_buffer_stream; + Argus::IBufferOutputStream * right_buffer_stream; + + ArgusStereoNode * node_; + }; + + auto module_descriptor = rcl_interfaces::msg::ParameterDescriptor{}; + module_descriptor.description = "Index specifying the stereo camera module"; + module_descriptor.read_only = false; + this->declare_parameter("module", 0, module_descriptor); + + auto sensor_descriptor = rcl_interfaces::msg::ParameterDescriptor{}; + sensor_descriptor.description = "Index specifying sensor mode for the stereo pair"; + sensor_descriptor.read_only = false; + this->declare_parameter("sensor", 0, sensor_descriptor); + + auto output_encoding_description = rcl_interfaces::msg::ParameterDescriptor{}; + output_encoding_description.description = "ROS image encoding to use for the output image"; + output_encoding_description.additional_constraints = + "Currently supported: 'rgb8' or 'mono8' (Default: 'mono8')"; + this->declare_parameter("output_encoding", std::string{"mono8"}, output_encoding_description); + + // + // Set up Argus API Framework, identify available camera modules and sensor modes, and create + // a capture session for the selected devices and corresponding sensor modes + // + + uint32_t module_index, sensor_counter, sensor_index; + + Argus::UniqueObj cameraProvider(Argus::CameraProvider::create()); + Argus::ICameraProvider * i_camera_provider = Argus::interface_cast( + cameraProvider); + if (!i_camera_provider) { + RCLCPP_ERROR(this->get_logger(), "Cannot get core camera provider interface\n"); + return; + } + RCLCPP_INFO(this->get_logger(), "Argus Version: %s\n", i_camera_provider->getVersion().c_str()); + + // Get camera devices vector and display available camera device models + std::vector camera_devices; + Argus::Status status = i_camera_provider->getCameraDevices(&camera_devices); + if (camera_devices.empty()) { + RCLCPP_ERROR(this->get_logger(), "Failed to get camera devices\n"); + return; + } + + if (camera_devices.size() < 2) { + RCLCPP_ERROR(this->get_logger(), "Must have at least two devices available\n"); + return; + } + + RCLCPP_INFO( + this->get_logger(), + "Set the stereo camera module using the node parameter \"module\""); + RCLCPP_INFO(this->get_logger(), "Set the sensor mode using the node parameter \"sensor\""); + RCLCPP_INFO( + this->get_logger(), + "Set the output image format using the node parameter \"output_encoding\". " + "Supported encodings: \"mono8\" and \"rgb8\" (Default: \"mono8\")\n"); + + ModuleInfo module_info[MAX_CAM_DEVICE]; + uint32_t module_count = 0; + memset(&module_info, 0, MAX_CAM_DEVICE * sizeof(ModuleInfo)); + + // cppcheck-suppress syntaxError + for (uint32_t i = 0; i < MAX_CAM_DEVICE; i++) { + module_info[i].initialized = false; + } + + char syncSensorId[MAX_MODULE_STRING]; + + // cppcheck-suppress syntaxError + for (uint32_t i = 0; i < camera_devices.size(); i++) { + Argus::ICameraProperties * camera_properties = Argus::interface_cast( + camera_devices[i]); + if (!camera_properties) { + RCLCPP_ERROR(this->get_logger(), "Failed to get ICameraProperties interface\n"); + return; + } + + const Argus::Ext::ISyncSensorCalibrationData * sync_sensor_calibration_data = + Argus::interface_cast(camera_devices[i]); + if (sync_sensor_calibration_data) { + sync_sensor_calibration_data->getSyncSensorModuleId(syncSensorId, sizeof(syncSensorId)); + + for (uint32_t j = 0; j <= module_count; j++) { + if (strcmp(syncSensorId, module_info[j].moduleName)) { + if (module_info[j].initialized == false) { + snprintf( + module_info[j].moduleName, sizeof(module_info[j].moduleName), "%s", + syncSensorId); + module_info[j].initialized = true; + module_info[j].camDevice[module_info[j].sensor_count++] = i; + } else { + continue; + } + + module_count++; + break; + } else { + module_info[j].camDevice[module_info[j].sensor_count++] = i; + break; + } + } + } else { + std::string model_string; + model_string = camera_properties->getModuleString(); + RCLCPP_INFO( + this->get_logger(), + "Failed to get ISyncSensorCalibrationData interface [1]\n"); + + for (uint32_t j = 0; j <= module_count; j++) { + if (strcmp(model_string.c_str(), module_info[j].moduleName)) { + if (module_info[j].initialized == false) { + snprintf( + module_info[j].moduleName, sizeof(module_info[j].moduleName), "%s", + model_string.c_str()); + module_info[j].initialized = true; + module_info[j].camDevice[module_info[j].sensor_count++] = i; + } else { + continue; + } + + module_count++; + break; + } else { + module_info[j].camDevice[module_info[j].sensor_count++] = i; + break; + } + } + } + } + + RCLCPP_INFO(this->get_logger(), "Following camera modules are available:"); + Argus::CameraDevice * camera_device; + Argus::ICameraProperties * properties; + Argus::Size2D res; + std::vector sensor_modes; + Argus::ISensorMode * sensor_properties; + + // Assuming both sensors of a stereo camera have same sensor modes + for (uint32_t i = 0; i < module_count; i++) { + RCLCPP_INFO( + this->get_logger(), "Module %d: %s module with %d sensors connected", i, + module_info[i].moduleName, module_info[i].sensor_count); + RCLCPP_INFO(this->get_logger(), "Available sensor modes for this module:"); + + camera_device = camera_devices[module_info[i].camDevice[0]]; + properties = Argus::interface_cast(camera_device); + status = properties->getAllSensorModes(&sensor_modes); + if (status != Argus::STATUS_OK) { + RCLCPP_ERROR(this->get_logger(), "Failed to get sensor modes vector\n"); + return; + } + sensor_counter = 0; + for (auto snsr : sensor_modes) { + sensor_properties = Argus::interface_cast(snsr); + res = sensor_properties->getResolution(); + RCLCPP_INFO(this->get_logger(), "%d (%d x %d)", sensor_counter++, res.width(), res.height()); + } + } + + Argus::UniqueObj request; + std::vector lr_cameras; + + // Get the parameter value for stereo camera module + this->get_parameter("module", module_index); + for (uint32_t j = 0; j < module_info[module_index].sensor_count; j++) { + lr_cameras.push_back(camera_devices[module_info[module_index].camDevice[j]]); + } + + module_info[module_index].capture_session = Argus::UniqueObj( + i_camera_provider->createCaptureSession( + lr_cameras)); + if (!module_info[module_index].capture_session) { + RCLCPP_ERROR(this->get_logger(), "Failed to create capture session\n"); + return; + } + Argus::ICaptureSession * capture_session = Argus::interface_cast( + module_info[module_index].capture_session); + if (!capture_session) { + RCLCPP_ERROR(this->get_logger(), "Failed to get ICaptureSession interface\n"); + return; + } + + module_info[module_index].stream_settings = Argus::UniqueObj( + capture_session->createOutputStreamSettings( + Argus::STREAM_TYPE_BUFFER)); + + Argus::IOutputStreamSettings * stream_settings = + Argus::interface_cast(module_info[module_index].stream_settings); + if (!stream_settings) { + RCLCPP_ERROR(this->get_logger(), "Cannot get IOutputStreamSettings interface\n"); + return; + } + + Argus::IBufferOutputStreamSettings * i_buffer_stream_settings = + Argus::interface_cast( + module_info[module_index].stream_settings); + if (!i_buffer_stream_settings) { + RCLCPP_ERROR(this->get_logger(), "Cannot get IBufferOutputStreamSettings interface\n"); + return; + } + + i_buffer_stream_settings->setBufferType(Argus::BUFFER_TYPE_EGL_IMAGE); + if (i_buffer_stream_settings->setSyncType(Argus::SYNC_TYPE_NONE) != Argus::STATUS_OK) { + RCLCPP_ERROR(this->get_logger(), "Failed to disable EGL sync"); + return; + } + + EGLint major, minor; + EGLDisplay egl_display = eglGetDisplay(EGL_DEFAULT_DISPLAY); + eglInitialize(egl_display, &major, &minor); + + for (uint32_t a = 0; a < module_info[module_index].sensor_count; a++) { + stream_settings->setCameraDevice(lr_cameras[a]); + module_info[module_index].stream[a] = Argus::UniqueObj( + capture_session->createOutputStream( + module_info[module_index].stream_settings.get())); + } + + // Streams for the capture session have been created + + // Get the parameter value for sensor model + this->get_parameter("sensor", sensor_index); + Argus::SensorMode * sensor_mode = sensor_modes[sensor_index]; + Argus::ISensorMode * iSensorMode = Argus::interface_cast(sensor_mode); + if (!iSensorMode) { + RCLCPP_ERROR(this->get_logger(), "Failed to get sensor mode interface\n"); + return; + } + + width_ = (iSensorMode->getResolution()).width(); + height_ = (iSensorMode->getResolution()).height(); + + Argus::Size2D size = + Argus::Size2D( + width_, + height_); + + ArgusSamples::NativeBuffer * native_buffers[MAX_CAM_DEVICE][NUM_BUFFERS]; + EGLImageKHR egl_images[MAX_CAM_DEVICE][NUM_BUFFERS]; + Argus::UniqueObj buffers[MAX_CAM_DEVICE][NUM_BUFFERS]; + + for (uint32_t a = 0; a < module_info[module_index].sensor_count; a++) { + Argus::IBufferOutputStream * i_buffer_output_stream = + Argus::interface_cast(module_info[module_index].stream[a]); + if (!i_buffer_output_stream) { + RCLCPP_INFO(this->get_logger(), "Failed to create IBufferOutputStream interface\n"); + } + + Argus::UniqueObj buffer_settings( + i_buffer_output_stream->createBufferSettings()); + if (!buffer_settings) { + RCLCPP_ERROR(this->get_logger(), "Failed to create buffer settings\n"); + return; + } + + Argus::IEGLImageBufferSettings * i_buffer_settings = + Argus::interface_cast(buffer_settings); + if (!i_buffer_settings) { + RCLCPP_ERROR(this->get_logger(), "Cannot get IEGLImageBufferSettings interface\n"); + return; + } + + i_buffer_settings->setEGLDisplay(egl_display); + + ArgusSamples::NvNativeBuffer * buf; + for (uint32_t j = 0; j < NUM_BUFFERS; j++) { + buf = ArgusSamples::NvNativeBuffer::create(size, NvBufferColorFormat_NV12); + native_buffers[a][j] = (ArgusSamples::NativeBuffer *)buf; + if (!native_buffers[a][j]) { + RCLCPP_ERROR(this->get_logger(), "Failed to create native buffer\n"); + return; + } + + egl_images[a][j] = native_buffers[a][j]->createEGLImage(egl_display); + if (egl_images[a][j] == EGL_NO_IMAGE_KHR) { + RCLCPP_ERROR(this->get_logger(), "Failed to create eglImage from native buffer\n"); + return; + } + + i_buffer_settings->setEGLImage(egl_images[a][j]); + + buffers[a][j].reset(i_buffer_output_stream->createBuffer(buffer_settings.get())); + + if (!Argus::interface_cast(buffers[a][j])) { + RCLCPP_ERROR(this->get_logger(), "Failed to create buffer\n"); + return; + } + if (i_buffer_output_stream->releaseBuffer(buffers[a][j].get()) != Argus::STATUS_OK) { + RCLCPP_ERROR(this->get_logger(), "Failed to release buffer for capture use\n"); + return; + } + } + } + + this->get_parameter("output_encoding", encoding_); + + module_info[module_index].sync_stereo_consumer = new SyncStereoConsumerThread( + &module_info[module_index], this); + + if (!(module_info[module_index].sync_stereo_consumer->initialize())) { + RCLCPP_ERROR(this->get_logger(), "Failed to initialize Consumer Thread\n"); + return; + } + if (!(module_info[module_index].sync_stereo_consumer->waitRunning())) { + RCLCPP_ERROR(this->get_logger(), "Consumer Thread does not enter running state\n"); + return; + } + + request = Argus::UniqueObj(capture_session->createRequest()); + Argus::IRequest * i_request = Argus::interface_cast(request); + if (!i_request) { + RCLCPP_ERROR(this->get_logger(), "Failed to get i_request interface\n"); + return; + } + + for (uint32_t a = 0; a < module_info[module_index].sensor_count; a++) { + i_request->enableOutputStream(module_info[module_index].stream[a].get()); + } + + for (uint32_t a = 0; a < module_info[module_index].sensor_count; a++) { + const Argus::Ext::ISyncSensorCalibrationData * sync_sensor_calibration_data = + Argus::interface_cast(lr_cameras[a]); + camerainfo_[a] = sensor_msgs::msg::CameraInfo::SharedPtr(new sensor_msgs::msg::CameraInfo()); + images_[a] = sensor_msgs::msg::Image::SharedPtr(new sensor_msgs::msg::Image()); + if (sync_sensor_calibration_data) { + SyncStereoCalibrationData(sync_sensor_calibration_data, camerainfo_[a]); + } else { + RCLCPP_INFO( + this->get_logger(), + "Failed to get ISyncSensorCalibrationData interface\n"); + } + } + + ComputeStereoRectificationData(camerainfo_); + + publishers_[0] = image_transport::create_camera_publisher(this, "/stereo/left/image_raw"); + publishers_[1] = image_transport::create_camera_publisher(this, "/stereo/right/image_raw"); + + Argus::ISourceSettings * source_settings = Argus::interface_cast(request); + if (!source_settings) { + RCLCPP_ERROR(this->get_logger(), "Failed to get source settings request interface\n"); + return; + } + + source_settings->setSensorMode(sensor_mode); + + // Submit capture for the specified time. + RCLCPP_INFO(this->get_logger(), "Starting repeat capture requests."); + if (capture_session->repeat(request.get()) != Argus::STATUS_OK) { + RCLCPP_ERROR(this->get_logger(), "Failed to start repeat capture request for preview"); + return; + } + while (rclcpp::ok()) { + } + + capture_session->stopRepeat(); + capture_session->waitForIdle(); + + for (uint32_t a = 0; a < module_info[module_index].sensor_count; a++) { + module_info[module_index].stream[a].reset(); + } + + RCLCPP_INFO(this->get_logger(), "Producer shutting down"); + for (uint32_t a = 0; a < module_info[module_index].sensor_count; a++) { + module_info[module_index].stream[a].reset(); + } + for (uint32_t a = 0; a < module_info[module_index].sensor_count; a++) { + for (uint32_t j = 0; j < NUM_BUFFERS; j++) { + buffers[a][j].reset(); + } + } + for (uint32_t a = 0; a < module_info[module_index].sensor_count; a++) { + for (uint32_t j = 0; j < NUM_BUFFERS; j++) { + delete native_buffers[a][j]; + } + } + module_info[module_index].capture_session.reset(); + module_info[module_index].stream_settings.reset(); + request.reset(); + cameraProvider.reset(); +} + +ArgusStereoNode::~ArgusStereoNode() +{ +} + +} // namespace argus +} // namespace isaac_ros + +// Register as a component +#include "rclcpp_components/register_node_macro.hpp" +RCLCPP_COMPONENTS_REGISTER_NODE(isaac_ros::argus::ArgusStereoNode) diff --git a/resources/100_library_left.JPG b/resources/100_library_left.JPG new file mode 100644 index 0000000..87e089e Binary files /dev/null and b/resources/100_library_left.JPG differ diff --git a/resources/100_library_right.JPG b/resources/100_library_right.JPG new file mode 100644 index 0000000..4c09e50 Binary files /dev/null and b/resources/100_library_right.JPG differ