Fixes #36814 - add content counts to capsule content info #913

ianballou · 2023-10-10T19:44:51Z

... and ...update-counts

Content counts for every indexed content type should be available.

To test:

Sync a smart proxy with different repository types
Run hammer capsule content info --id <id>
See something like the following:

Lifecycle Environments: 
 1) Name:          Precipitation
    Organization:  Default Organization
    Content Views: 
     1) Name:           Zoo View
        Composite:      no
        Last Published: 2023/10/09 19:18:15
        Repositories:   
         1) Repository ID:   2
            Repository Name: Zoo
            Content Counts:  
                Packages: 32
                Errata:   4
     2) Name:           Everything View
        Composite:      no
        Last Published: 2023/10/10 15:59:58
        Repositories:   
         1) Repository ID:   41
            Repository Name: Zoo
            Content Counts:  
                Packages: 32
                Errata:   4
         2) Repository ID:   40
            Repository Name: Debian Repo
            Content Counts:  
                Debian Packages: 8
         3) Repository ID:   42
            Repository Name: OSTree Repo
            Content Counts:  
                OSTree Refs: 2
         4) Repository ID:   44
            Repository Name: prometheus/busybox
            Content Counts:  
                Container Tags:           3
                Container Manifests:      9
                Container Manifest Lists: 3
         5) Repository ID:   47
            Repository Name: File Repo
            Content Counts:  
                Files: 250
         6) Repository ID:   46
            Repository Name: ansible repo
            Content Counts:  
                Ansible Collections: 446
         7) Repository ID:   45
            Repository Name: Python Repo
            Content Counts:  
                Python Packages: 2
         8) Repository ID:   43
            Repository Name: EPEL 8 Modular x86_64
            Content Counts:  
                Packages:       290
                Module Streams: 24
                Errata:         14

Run hammer capsule content update-counts --id <id>
Hammer should either follow the counts update task or return the task info when providing --async.

theforeman-bot · 2023-10-10T19:44:55Z

Issues: #36814

ianballou · 2023-10-10T19:45:10Z

Requires this Katello PR: Katello/katello#10765

ianballou · 2023-10-10T20:25:03Z

The test failures appear to be occurring against master as well.

Looks like the failures are deprecation warnings from Mocha.

chris1984 · 2023-10-10T20:46:29Z

The test failures appear to be occurring against master as well.

Looks like the failures are deprecation warnings from Mocha.

I'll take a look tomorrow and see what is going on. I swear the wind blows and the test fails lol.

chris1984 · 2023-10-11T14:45:54Z

@ianballou It looks like this test is failing:

  1) Failure:
capsule content synchronization_status::output#test_0004_shows environments that can be synchronized [/home/runner/work/hammer-cli-katello/hammer-cli-katello/test/functional/capsule/content/synchronization_status_test.rb:52]:
Regex /Status:[ ]+1\ environment\(s\)\ can\ be\ synchronized:\ Library/ didn't match the output:
Last sync:                    2016/01/10 00:27:51
Status:                       Capsule is synchronized
Currently running sync tasks: 

Last failure:                 
    Task id:  
    Messages:

I am guessing the changes you made to the sync status command, made it unhappy

ianballou · 2023-10-12T14:50:23Z

Tests should be fixed now!

chris1984 · 2023-10-12T15:52:55Z

Code looks fine, starting to test

chris1984 · 2023-10-12T19:13:30Z

update-counts works correctly:

[vagrant@centos7-hammer-devel ~]$ hammer capsule content update-counts --id 2
[..................................................................................................................................................................................] [100%]
[vagrant@centos7-hammer-devel ~]$ hammer capsule content update-counts --id 2 --async
Capsule content counts are being updated in task c0447907-420d-41dd-9393-869f5f8dbff8.

When I try to do the hammer capsule content info --id 2 I am getting Error: undefined method each' for nil:NilClass`

NoMethodError (undefined method `each' for nil:NilClass):
    /home/vagrant/hammer-cli-katello/lib/hammer_cli_katello/capsule.rb:219:in `block (2 levels) in extend_data'
    /home/vagrant/hammer-cli-katello/lib/hammer_cli_katello/capsule.rb:218:in `each'
    /home/vagrant/hammer-cli-katello/lib/hammer_cli_katello/capsule.rb:218:in `block in extend_data'
    /home/vagrant/hammer-cli-katello/lib/hammer_cli_katello/capsule.rb:217:in `each'
    /home/vagrant/hammer-cli-katello/lib/hammer_cli_katello/capsule.rb:217:in `extend_data'
    /home/vagrant/hammer-cli-foreman/lib/hammer_cli_foreman/commands.rb:503:in `send_request'
    /home/vagrant/hammer-cli/lib/hammer_cli/apipie/command.rb:34:in `execute'
    /home/vagrant/.gem/ruby/gems/clamp-1.3.2/lib/clamp/command.rb:66:in `run'

I don't see any traceback on the production.log

ianballou · 2023-10-12T20:58:14Z

When I try to do the hammer capsule content info --id 2 I am getting Error: undefined method each' for nil:NilClass`

@chris1984 can I see your debug output? Wondering if you have a CV without any repositories. I don't think I tried that.

chris1984 · 2023-10-16T14:49:26Z

@ianballou Here is the debug output:

hammer -d capsule content info --id 2
[ INFO 2023-10-16T14:48:38 API] Server: https://centos8-katello-devel.area51.example.com
[ INFO 2023-10-16T14:48:38 API] GET /katello/api/capsules/2/content/sync
[DEBUG 2023-10-16T14:48:38 API] Params: {}
[DEBUG 2023-10-16T14:48:38 API] Headers: {
    :params => {}
}
[DEBUG 2023-10-16T14:48:38 API] Using authenticator: HammerCLIForeman::Api::InteractiveBasicAuth
[DEBUG 2023-10-16T14:48:40 API] Response: {
              "last_sync_time" => "2023-10-12 18:52:32 UTC",
             "download_policy" => "inherit",
           "active_sync_tasks" => [],
      "last_failed_sync_tasks" => [],
      "lifecycle_environments" => [
        [0] {
                    "library" => false,
                         "id" => 2,
                       "name" => "Foo",
                      "label" => "Foo",
                "description" => nil,
            "organization_id" => 1,
               "organization" => {
                 "name" => "Default Organization",
                "label" => "Default_Organization",
                   "id" => 1
            },
                   "syncable" => false,
                     "counts" => {
                "content_views" => 1
            },
              "content_views" => [
                [0] {
                                "id" => 2,
                             "label" => "Zoo",
                              "name" => "Zoo",
                         "composite" => false,
                    "last_published" => "2023-10-12 18:50:47 UTC",
                           "default" => false,
                        "up_to_date" => true,
                            "counts" => {
                        "repositories" => 7
                    },
                    "content_counts" => {
                        "content_view_versions" => {
                            "2" => {
                                             "repositories" => {
                                    "24" => {
                                        "file" => 3
                                    },
                                    "25" => {
                                        "ansible.collection_version" => 14
                                    },
                                    "26" => {
                                                         "rpm" => 35,
                                                     "erratum" => 4,
                                               "package_group" => 2,
                                         "rpm.packagecategory" => 1,
                                        "rpm.packagelangpacks" => 1
                                    },
                                    "27" => {
                                            "ostree_ref" => 2,
                                         "ostree.commit" => 2,
                                         "ostree.config" => 1,
                                         "ostree.object" => 3,
                                        "ostree.summary" => 1
                                    },
                                    "28" => {
                                        "python_package" => 2
                                    },
                                    "29" => {
                                                          "rpm" => 22,
                                                      "erratum" => 7,
                                                "module_stream" => 14,
                                                "package_group" => 2,
                                          "rpm.packagecategory" => 1,
                                        "rpm.distribution_tree" => 1,
                                        "rpm.modulemd_defaults" => 3
                                    },
                                    "30" => {
                                                  "docker_tag" => 3,
                                              "container.blob" => 27,
                                             "docker_manifest" => 9,
                                        "docker_manifest_list" => 3
                                    }
                                },
                                "cv_version_content_counts" => {
                                                           "rpm" => 57,
                                                          "file" => 3,
                                                       "erratum" => 11,
                                                    "docker_tag" => 3,
                                                    "ostree_ref" => 2,
                                                 "module_stream" => 14,
                                                 "ostree.commit" => 2,
                                                 "ostree.config" => 1,
                                                 "ostree.object" => 3,
                                                 "package_group" => 4,
                                                "container.blob" => 27,
                                                "ostree.summary" => 1,
                                                "python_package" => 2,
                                               "docker_manifest" => 9,
                                           "rpm.packagecategory" => 2,
                                          "docker_manifest_list" => 3,
                                          "rpm.packagelangpacks" => 1,
                                         "rpm.distribution_tree" => 1,
                                         "rpm.modulemd_defaults" => 3,
                                    "ansible.collection_version" => 14
                                }
                            }
                        }
                    }
                }
            ]
        }
    ],
    "unsyncable_content_types" => []
}

ianballou · 2023-10-16T15:50:40Z

Ah @chris1984 I think you may be missing this PR: Katello/katello#10745

ianballou · 2023-10-19T17:27:15Z

One corner case I need to check is what happens if two environments share one repository, and both envs are linked to the smart proxy, but only one is actually synced.

chris1984 · 2023-10-19T17:29:13Z

One corner case I need to check is what happens if two environments share one repository, and both envs are linked to the smart proxy, but only one is actually synced.

I had to reinstall my smart-proxy, did you want to test and see if this needs an update before I test it again? I was going to test it today. Since yesterday was a dumpster fire with meetings etc.

ianballou · 2023-10-20T18:24:54Z

I'm starting to build my code off of Samir's Katello PR: Katello/katello#10766

It stops errors from occurring when calculating smart proxy counts for LCEs that haven't been synced yet.

Here's an example for what I have currently:

 2) Name:          Production
    Organization:  Default Organization
    Content Views: 
     1) Name:           Zoo View
        Composite:      no
        Last Published: 2023/10/20 17:59:16
        Repositories:   
         1) Repository ID:   63
            Repository Name: Python Repo
            Content Counts:  
                Warning: Content view must be synced to see content counts
         2) Repository ID:   62
            Repository Name: Zoo
            Content Counts:  
                Warning: Content view must be synced to see content counts

There is a warning if the repository isn't up to date on the smart proxy.

* Add capsule content update-counts command

ianballou · 2023-10-23T21:27:17Z

@chris1984 this is ready for another review. Grab @sjha4 's other PR first: Katello/katello#10766

An extra thing to test is the warning that pops up for repositories in LCEs that haven't been synced yet. It's easier to test this if you disable the setting that syncs capsules after promotion.

Example:

 5) Name:          Mega Production
    Organization:  Default Organization
    Content Views: 
     1) Name:           Zoo View
        Composite:      no
        Last Published: 2023/10/20 17:59:16
        Repositories:   
         1) Repository ID:   63
            Repository Name: Python Repo
            Content Counts:  
                Warning: Content view must be synced to see content counts
         2) Repository ID:   62
            Repository Name: Zoo
            Content Counts:  
                Warning: Content view must be synced to see content counts

chris1984

First part works and I get the warning :)

Lifecycle Environments: 
 1) Name:          Capsule
    Organization:  Default Organization
    Content Views: 
     1) Name:           Everything
        Composite:      no
        Last Published: 2023/10/24 17:47:40
        Repositories:   
         1) Repository ID:   9
            Repository Name: Dockah
            Content Counts:  
                Warning: Content view must be synced to see content counts
         2) Repository ID:   13
            Repository Name: Partha-Modules
            Content Counts:  
                Warning: Content view must be synced to see content counts
         3) Repository ID:   12
            Repository Name: Medium
            Content Counts:  
                Warning: Content view must be synced to see content counts
         4) Repository ID:   11
            Repository Name: OSTree-James
            Content Counts:  
                Warning: Content view must be synced to see content counts
         5) Repository ID:   10
            Repository Name: Files
            Content Counts:  
                Warning: Content view must be synced to see content counts
         6) Repository ID:   15
            Repository Name: Ansible
            Content Counts:  
                Warning: Content view must be synced to see content counts
         7) Repository ID:   14
            Repository Name: Python
            Content Counts:  
                Warning: Content view must be synced to see content counts
         8) Repository ID:   16
            Repository Name: Debian
            Content Counts:  
                Warning: Content view must be synced to see content counts

After syncing I can get see everything:

Lifecycle Environments: 
 1) Name:          Capsule
    Organization:  Default Organization
    Content Views: 
     1) Name:           Everything
        Composite:      no
        Last Published: 2023/10/24 17:47:40
        Repositories:   
         1) Repository ID:   9
            Repository Name: Dockah
            Content Counts:  
                Container Tags:           3
                Container Manifests:      9
                Container Manifest Lists: 3
         2) Repository ID:   13
            Repository Name: Partha-Modules
            Content Counts:  
                Packages:       22
                Module Streams: 14
                Package Groups: 2
                Errata:         7
         3) Repository ID:   12
            Repository Name: Medium
            Content Counts:  
                Packages: 403
                Errata:   2
         4) Repository ID:   11
            Repository Name: OSTree-James
            Content Counts:  
                OSTree Refs: 2
         5) Repository ID:   10
            Repository Name: Files
            Content Counts:  
                Files: 3
         6) Repository ID:   15
            Repository Name: Ansible
            Content Counts:  
                Ansible Collections: 14
         7) Repository ID:   14
            Repository Name: Python
            Content Counts:  
                Python Packages: 2
         8) Repository ID:   16
            Repository Name: Debian
            Content Counts:  
                Debian Packages: 5

ACK nice work @ianballou

Let me know if you need me to merge it if you don't have perms, otherwise ship it

pr-processor bot added the Not yet reviewed label Oct 10, 2023

ianballou force-pushed the 36814-proxy-content-counts branch from aef0835 to ac95e6a Compare October 10, 2023 19:54

ianballou force-pushed the 36814-proxy-content-counts branch from ac95e6a to c35ecc7 Compare October 12, 2023 14:50

ianballou force-pushed the 36814-proxy-content-counts branch 2 times, most recently from efbae78 to 0afb3a7 Compare October 23, 2023 21:03

Fixes #36814 - add content counts to capsule content info

614b41d

* Add capsule content update-counts command

ianballou force-pushed the 36814-proxy-content-counts branch from 0afb3a7 to 614b41d Compare October 23, 2023 21:21

chris1984 approved these changes Oct 24, 2023

View reviewed changes

pr-processor bot removed the Not yet reviewed label Oct 24, 2023

chris1984 self-assigned this Oct 24, 2023

chris1984 added Ready for merge Requires Katello PR labels Oct 24, 2023

chris1984 merged commit f3c33cb into Katello:master Oct 25, 2023
3 checks passed

vsedmik mentioned this pull request Nov 22, 2023

Add CLI tests for Capsule Content Counts SatelliteQE/robottelo#13163

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes #36814 - add content counts to capsule content info #913

Fixes #36814 - add content counts to capsule content info #913

ianballou commented Oct 10, 2023

theforeman-bot commented Oct 10, 2023

ianballou commented Oct 10, 2023

ianballou commented Oct 10, 2023 •

edited

Loading

chris1984 commented Oct 10, 2023 •

edited

Loading

chris1984 commented Oct 11, 2023 •

edited

Loading

ianballou commented Oct 12, 2023

chris1984 commented Oct 12, 2023

chris1984 commented Oct 12, 2023

ianballou commented Oct 12, 2023

chris1984 commented Oct 16, 2023 •

edited

Loading

ianballou commented Oct 16, 2023

ianballou commented Oct 19, 2023

chris1984 commented Oct 19, 2023 •

edited

Loading

ianballou commented Oct 20, 2023

ianballou commented Oct 23, 2023 •

edited

Loading

chris1984 left a comment •

edited

Loading

Fixes #36814 - add content counts to capsule content info #913

Fixes #36814 - add content counts to capsule content info #913

Conversation

ianballou commented Oct 10, 2023

theforeman-bot commented Oct 10, 2023

ianballou commented Oct 10, 2023

ianballou commented Oct 10, 2023 • edited Loading

chris1984 commented Oct 10, 2023 • edited Loading

chris1984 commented Oct 11, 2023 • edited Loading

ianballou commented Oct 12, 2023

chris1984 commented Oct 12, 2023

chris1984 commented Oct 12, 2023

ianballou commented Oct 12, 2023

chris1984 commented Oct 16, 2023 • edited Loading

ianballou commented Oct 16, 2023

ianballou commented Oct 19, 2023

chris1984 commented Oct 19, 2023 • edited Loading

ianballou commented Oct 20, 2023

ianballou commented Oct 23, 2023 • edited Loading

chris1984 left a comment • edited Loading

Choose a reason for hiding this comment

ianballou commented Oct 10, 2023 •

edited

Loading

chris1984 commented Oct 10, 2023 •

edited

Loading

chris1984 commented Oct 11, 2023 •

edited

Loading

chris1984 commented Oct 16, 2023 •

edited

Loading

chris1984 commented Oct 19, 2023 •

edited

Loading

ianballou commented Oct 23, 2023 •

edited

Loading

chris1984 left a comment •

edited

Loading