Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sample(storage): Soft deleted Bucket Restore #28760

Draft
wants to merge 71 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
71 commits
Select commit Hold shift + click to select a range
0f5b378
WIP- able to fetch generation and list of soft deleted bucket
shubhangi-google Nov 25, 2024
c13c271
wip- fetch soft deleted bucket and restore bucket
shubhangi-google Nov 27, 2024
703a74f
refactor
shubhangi-google Nov 28, 2024
113e074
adding unit tests
shubhangi-google Dec 11, 2024
d8389ee
refactor
shubhangi-google Dec 12, 2024
4805dcf
update
shubhangi-google Dec 12, 2024
4628b57
fixing lint issues
shubhangi-google Dec 13, 2024
331713a
refactor
shubhangi-google Dec 13, 2024
7bb513c
removing unwanted changes
shubhangi-google Dec 16, 2024
fd595ed
removing commented code
shubhangi-google Dec 16, 2024
5876b21
removing unwanted file change
shubhangi-google Dec 16, 2024
8bb3590
refactor
shubhangi-google Dec 16, 2024
0c7316a
refactor
shubhangi-google Dec 18, 2024
dc22c71
WIP-samples
shubhangi-google Dec 12, 2024
45c0bc7
adding samples and tests
shubhangi-google Dec 17, 2024
2e96036
refactor
shubhangi-google Dec 18, 2024
3728dcf
refactor
shubhangi-google Dec 19, 2024
4265f54
updating comments
shubhangi-google Dec 19, 2024
3c89fef
fix lint issues
shubhangi-google Dec 20, 2024
d3e3db6
fix lint issues
shubhangi-google Dec 20, 2024
36fbf02
fix review comments
shubhangi-google Dec 21, 2024
69a242b
removing pry
shubhangi-google Dec 21, 2024
46b987e
fix lint
shubhangi-google Dec 21, 2024
0d902d5
try cli issue fix
shubhangi-google Dec 23, 2024
b187c21
fix lint
shubhangi-google Dec 23, 2024
b608825
removing unwanted code
shubhangi-google Jan 15, 2025
cc5dd1e
adding check to see if bucket is deleted or not
shubhangi-google Jan 15, 2025
6700876
debugging
shubhangi-google Jan 15, 2025
9538c91
debugging
shubhangi-google Jan 15, 2025
ae9e2af
debugging
shubhangi-google Jan 15, 2025
dbdbe58
debugging
shubhangi-google Jan 15, 2025
d67ebf4
debugging
shubhangi-google Jan 15, 2025
f30c889
debugging
shubhangi-google Jan 15, 2025
8040fab
debugging
shubhangi-google Jan 15, 2025
88438a3
updating gemspec
shubhangi-google Jan 16, 2025
70236f7
debugging
shubhangi-google Jan 17, 2025
b2f67ad
debugging
shubhangi-google Jan 17, 2025
6108636
undo adding static email
shubhangi-google Jan 17, 2025
04a56f1
updating gemspec -- debugging
shubhangi-google Jan 17, 2025
dcb47e7
updating gem spec -- debugging
shubhangi-google Jan 17, 2025
360f07a
debugging
shubhangi-google Jan 17, 2025
f42e11d
debugging
shubhangi-google Jan 20, 2025
49834d4
debugging
shubhangi-google Jan 20, 2025
9b023d0
addressing nit comments
shubhangi-google Jan 22, 2025
2a7f238
debugging
shubhangi-google Jan 22, 2025
63c4106
debugging
shubhangi-google Jan 22, 2025
c7a2e76
debugging
shubhangi-google Jan 23, 2025
9191faf
debugging
shubhangi-google Jan 23, 2025
6395dc5
debugging
shubhangi-google Jan 23, 2025
4feec6c
debugging
shubhangi-google Jan 23, 2025
39edb70
debugging
shubhangi-google Jan 23, 2025
25119ab
debugging
shubhangi-google Jan 23, 2025
6af39e7
debugging
shubhangi-google Jan 23, 2025
63b8485
debugging
shubhangi-google Jan 23, 2025
0bc844f
debugging
shubhangi-google Jan 23, 2025
a442451
debugging
shubhangi-google Jan 23, 2025
44b3aae
debugging
shubhangi-google Jan 23, 2025
5ba2762
debugging
shubhangi-google Jan 23, 2025
e7dcbce
debugging
shubhangi-google Jan 23, 2025
4838eb5
debugging
shubhangi-google Jan 23, 2025
2edc311
debugging
shubhangi-google Jan 23, 2025
7a6f719
debugging
shubhangi-google Jan 23, 2025
199acc5
debugging
shubhangi-google Jan 24, 2025
120d90c
debugging
shubhangi-google Jan 27, 2025
2a48db3
resolving lint issues
shubhangi-google Jan 27, 2025
a559a06
resolving lint issue
shubhangi-google Jan 27, 2025
4bdeb8b
lint issue
shubhangi-google Jan 27, 2025
da04abb
Merge branch 'main' into soft_deleted_bucket_restore_with_sample
shubhangi-google Feb 13, 2025
0b10a6a
Merge branch 'main' into soft_deleted_bucket_restore_with_sample
shubhangi-google Feb 13, 2025
4172e24
adding retention_period to sample bucket
shubhangi-google Feb 14, 2025
33b3f07
Merge remote-tracking branch 'refs/remotes/origin/soft_deleted_bucket…
shubhangi-google Feb 14, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions google-cloud-storage/samples/acceptance/buckets_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,10 @@
require_relative "../storage_get_public_access_prevention"
require_relative "../storage_get_requester_pays_status"
require_relative "../storage_get_retention_policy"
require_relative "../storage_get_soft_deleted_bucket"
require_relative "../storage_get_uniform_bucket_level_access"
require_relative "../storage_list_buckets"
require_relative "../storage_list_soft_deleted_buckets"
require_relative "../storage_lock_retention_policy"
require_relative "../storage_remove_bucket_label"
require_relative "../storage_remove_cors_configuration"
Expand Down Expand Up @@ -119,6 +121,45 @@
end
end

describe "storage_soft_deleted_bucket" do
let(:new_bucket_name) { random_bucket_name }

it "get soft deleted bucket, its soft_delete_time and hard_delete_time" do
new_bucket = storage_client.create_bucket new_bucket_name
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this new bucket have soft_delete_policy ?
If not, we won't be able to recover the bucket.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bajajneha27 on console I can see this for one of the ruby sample test bucket
image

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the soft_delete_policy set by default if you create a new bucket?

If not, you'll have to pass it while creating the new bucket.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes this is set by default

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm. Can you explicitly check the policy in test too, so that we're sure we've the policy in place.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bajajneha27 tried to set retention period for the bucket explicitly but getting same result
also I can see in this doc (https://cloud.google.com/storage/docs/soft-delete#retention-duration) that default soft deleted policy is set as 7 days for a bucket

new_bucket.retention_period = retention_period
new_generation = new_bucket.generation
puts storage_client.service_account_email
puts new_bucket.policy.roles

# ensuring bucket is created
assert new_bucket.exists?, "Bucket #{new_bucket_name} should exist"

# fetching bucket
check_bucket = storage_client.bucket new_bucket_name
puts "new bucket name-- #{check_bucket.name}"
puts "new bucket generation-- #{check_bucket.generation}"
puts "new bucket retention period-- #{check_bucket.retention_period}"

delete_bucket_helper new_bucket_name
# Check if the bucket is deleted
deleted_bucket = storage_client.bucket new_bucket_name
refute deleted_bucket, "Bucket #{new_bucket_name} should not exist"

# fetching a soft deleted bucket
output, _err = capture_io do
get_soft_deleted_bucket bucket_name: new_bucket_name, generation: new_generation
end
assert_includes output, "soft_delete_time for #{new_bucket_name} is"
end

it "lists soft deleted buckets" do
list_deleted_bucket, _err = capture_io do
list_soft_deleted_buckets
end
assert list_deleted_bucket, "List of soft deleted bucket should not be blank"
end
end

describe "storage_create_bucket_dual_region" do
it "creates dual region bucket" do
location = "US"
Expand Down
17 changes: 17 additions & 0 deletions google-cloud-storage/samples/acceptance/project_test.rb
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,13 @@

require_relative "helper"
require_relative "../storage_get_service_account"
require_relative "../storage_restore_bucket"

describe "Storage Quickstart" do
let(:project) { Google::Cloud::Storage.new }
let(:bucket) { fixture_bucket }
let(:generation) { bucket.generation }
let(:new_bucket_name) { random_bucket_name }

it "get_service_account" do
email = nil
Expand All @@ -28,3 +32,16 @@
assert_includes out, "@gs-project-accounts.iam.gserviceaccount.com"
end
end

describe "storage_soft_deleted_bucket" do
let(:generation) { bucket.generation }
let(:bucket) { fixture_bucket }

it "restores a soft deleted bucket" do
delete_bucket_helper bucket.name
_out, _err = capture_io do
restore_bucket bucket_name: bucket.name, generation: generation
end
assert "#{bucket.name} Bucket restored"
end
end
41 changes: 41 additions & 0 deletions google-cloud-storage/samples/storage_get_soft_deleted_bucket.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# [START storage_get_soft_deleted_bucket]
def get_soft_deleted_bucket bucket_name:, generation:
# The ID of your GCS bucket
# bucket_name = "your-unique-bucket-name"
# The generation no of your GCS bucket
# generation = "1234567896987"

require "google/cloud/storage"

storage = Google::Cloud::Storage.new
# fetching soft deleted bucket with soft_delete_time and hard_delete_time
deleted_bucket_fetch = storage.bucket bucket_name, generation: generation, soft_deleted: true

soft_delete_time = deleted_bucket_fetch.soft_delete_time
hard_delete_time = deleted_bucket_fetch.hard_delete_time

if (soft_delete_time && hard_delete_time).nil?
puts "Not Found"
else
puts "soft_delete_time for #{deleted_bucket_fetch.name} is - #{soft_delete_time}"
puts "hard_delete_time for #{deleted_bucket_fetch.name} is - #{hard_delete_time}"
end
end
# [END storage_get_soft_deleted_bucket]


get_soft_deleted_bucket bucket_name: ARGV.shift, generation: ARGV.shift if $PROGRAM_NAME == __FILE__
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# [START storage_list_soft_deleted_buckets]
def list_soft_deleted_buckets
require "google/cloud/storage"

storage = Google::Cloud::Storage.new

# fetching soft deleted bucket list
deleted_buckets = storage.buckets soft_deleted: true

deleted_buckets.each do |bucket|
puts bucket.name
end
end
# [END storage_list_soft_deleted_buckets]

list_soft_deleted_buckets if $PROGRAM_NAME == __FILE__
36 changes: 36 additions & 0 deletions google-cloud-storage/samples/storage_restore_bucket.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# [START storage_restore_bucket]
def restore_bucket bucket_name:, generation:
# The ID of your GCS bucket
# bucket_name = "your-unique-bucket-name"
# The generation no of your GCS bucket
# generation = "1234567896987"

require "google/cloud/storage"

storage = Google::Cloud::Storage.new

bucket_restored = storage.restore_bucket bucket_name, generation

if bucket_restored.name == bucket_name
puts "#{bucket_name} Bucket restored"
else
puts "#{bucket_name} Bucket not restored"
end
end
# [END storage_restore_bucket]

restore_bucket bucket_name: ARGV.shift, generation: ARGV.shift if $PROGRAM_NAME == __FILE__