Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modularize Block Factory #8499

Merged
merged 4 commits into from
Jan 16, 2025
Merged

Modularize Block Factory #8499

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
72a60d7
Modularize Block Factory
itaigilo Jan 15, 2025
a184059
Import lakeFS module
itaigilo Jan 15, 2025
6dc4bd1
Merge branch 'master' into feature/modularize-block-factory
itaigilo Jan 15, 2025
6f3e08f
Rename imports
itaigilo Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cmd/lakefs/cmd/run.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ import (
"github.com/go-co-op/gocron"
"github.com/spf13/cobra"
"github.com/spf13/viper"
"github.com/treeverse/lakefs/modules/block/factory"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing I forgot!
Let's explicitly give it a name to avoid collisions in future factories:

Suggested change
"github.com/treeverse/lakefs/modules/block/factory"
blockfactory "github.com/treeverse/lakefs/modules/block/factory"

"github.com/treeverse/lakefs/pkg/actions"
"github.com/treeverse/lakefs/pkg/api"
"github.com/treeverse/lakefs/pkg/auth"
Expand All @@ -26,7 +27,6 @@ import (
authremote "github.com/treeverse/lakefs/pkg/auth/remoteauthenticator"
"github.com/treeverse/lakefs/pkg/authentication"
"github.com/treeverse/lakefs/pkg/block"
"github.com/treeverse/lakefs/pkg/block/factory"
"github.com/treeverse/lakefs/pkg/catalog"
"github.com/treeverse/lakefs/pkg/config"
"github.com/treeverse/lakefs/pkg/gateway"
Expand Down
1 change: 1 addition & 0 deletions go.work
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,6 @@ go 1.23

use (
.
./modules/block/factory
./webui
)
19 changes: 19 additions & 0 deletions modules/block/factory/build.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
package factory

import (
"context"

"github.com/treeverse/lakefs/pkg/block"
"github.com/treeverse/lakefs/pkg/block/factory"
"github.com/treeverse/lakefs/pkg/block/params"
"github.com/treeverse/lakefs/pkg/stats"
)

func BuildBlockAdapter(ctx context.Context, statsCollector stats.Collector, c params.AdapterConfig) (block.Adapter, error) {
adapter, err := factory.BuildBlockAdapter(ctx, statsCollector, c)
if err != nil {
return nil, err
}

return block.NewMetricsAdapter(adapter), nil
}
5 changes: 5 additions & 0 deletions modules/block/factory/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
module github.com/treeverse/lakefs/modules/block/factory

go 1.23

// This module uses the go.work file to get all package dependencies from lakefs
5 changes: 5 additions & 0 deletions modules/block/factory/go.work
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
go 1.23

use (
.
)
Copy link
Member

@N-o-Z N-o-Z Jan 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are importing the lakefs package to the factory module - you need to define it in the work file so it will use the local copy

Suggested change
use (
.
)
use (
.
../../..
)

9 changes: 0 additions & 9 deletions pkg/block/factory/build.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,15 +27,6 @@ const (
)

func BuildBlockAdapter(ctx context.Context, statsCollector stats.Collector, c params.AdapterConfig) (block.Adapter, error) {
adapter, err := buildBlockAdapter(ctx, statsCollector, c)
if err != nil {
return nil, err
}

return block.NewMetricsAdapter(adapter), nil
}

func buildBlockAdapter(ctx context.Context, statsCollector stats.Collector, c params.AdapterConfig) (block.Adapter, error) {
blockstore := strings.ToLower(c.BlockstoreType())
logging.FromContext(ctx).
WithField("type", blockstore).
Expand Down
2 changes: 1 addition & 1 deletion pkg/catalog/catalog.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ import (
"github.com/hashicorp/go-multierror"
lru "github.com/hnlq715/golang-lru"
"github.com/rs/xid"
"github.com/treeverse/lakefs/modules/block/factory"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The run file already imports and instantiate the block adapter, however catalog also instantiates it's own block adapter, this is probably for historical reasons.
Consider - passing the block adapter in run to the catalog New function

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I get what you're suggesting -
In the run file, the BuildBlockAdapter is initialized with statsCollector, while in the catalog it isn't, no?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"github.com/treeverse/lakefs/modules/block/factory"
blockfactory "github.com/treeverse/lakefs/modules/block/factory"

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's say this happens, and we have both imports (which we currently don't) -
Why the modules one should be blockfactory and the other just factory?

First, I don't think such convention is currently necessary.
And if it is necessary - please have a convention that reflects the fact that it's a module. I'd say moduleblockfactory, but it's too much.

Bottom line, I think we should leave it for now, since it doesn't make things clearer IMO.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you mean lets say it happens - we're going to have a module called config/factory when you import both, they will both be factory. One of the will be automatically renamed but I believe the convention should be to explicitly name them!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahhh, I thought you meant a module and a local with the same name.
Agree.
Updated.

"github.com/treeverse/lakefs/pkg/batch"
"github.com/treeverse/lakefs/pkg/block"
"github.com/treeverse/lakefs/pkg/block/factory"
"github.com/treeverse/lakefs/pkg/config"
"github.com/treeverse/lakefs/pkg/graveler"
"github.com/treeverse/lakefs/pkg/graveler/branch"
Expand Down
2 changes: 1 addition & 1 deletion pkg/config/config_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ import (
awsconfig "github.com/aws/aws-sdk-go-v2/config"
"github.com/go-test/deep"
"github.com/spf13/viper"
"github.com/treeverse/lakefs/modules/block/factory"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"github.com/treeverse/lakefs/modules/block/factory"
blockfactory "github.com/treeverse/lakefs/modules/block/factory"

"github.com/treeverse/lakefs/pkg/block"
"github.com/treeverse/lakefs/pkg/block/factory"
"github.com/treeverse/lakefs/pkg/block/gs"
"github.com/treeverse/lakefs/pkg/block/local"
"github.com/treeverse/lakefs/pkg/config"
Expand Down
Loading