Are you using Rudolfs in production? If so, drop me a note!

[jasonwhite]

I would love to know who has a stake in this project. I'm pretty confident this is the best LFS server in the universe (even compared to commercial products). For open source projects it is really hard to tell who is using it until something breaks. There is no telemetry to give me usage statistics (and I have no plans to add anything like that), so I rely on folks to tell me.

If you are using Rudolfs in production and care about this project, please let me know. If you don't feel comfortable leaving a reply in this discussion, you can also email me instead (see my GitHub profile page for my email address). You don't need to mention your company name if you don't want to. I just want to get a feel for how many people rely on this.

[rudoi]

Hi Jason!

The Believer Company is a small game studio using LFS to store assets from our Unreal Engine projects. We initially had a private fork of Rudolfs on top of which we implemented a few things, but we've since started our own GitHub org and placed the repo there at https://github.com/believer-oss/lfs-rs. We renamed it for a bit of a silly reason: it looked like it was named after me (my GH handle is my last name).

We made a judgment call that some of the items we implemented made our repo not upstream-able, for example we have GitHub-based auth built into the server. We also have implemented LFS locking that uses DynamoDB as its store. If you feel like any of this should go upstream, let me know - we'll do the work.

All that said, we feel like we do count as "using Rudolfs in production" - we evaluated many LFS server options and Rudolfs is indeed the best one that we came across.

[jasonwhite]

Glad you have found this project useful. Those are some great changes!

Both the GitHub-based authentication and the file locking would be great to upstream. Those are probably the two biggest missing features currently. Although, I think they should both be off-unless-configured to maintain backwards compatibility (and because not everyone uses GitHub and DynamoDB).

A couple of notes relevant to each feature:

1. GitHub-based authentication: Not everyone uses GitHub of course. Additionally, the LFS server <org>/<project>/ URL paths don't necessarily need to match the equivalent Git project URL paths. It can sometimes be useful to have multiple Git projects share the same LFS <org>/<project> if there are many LFS objects in common.
2. File locking via DynamoDB: Using a central DB for this seems like the right approach. I had thought to use S3 for this before, but S3 is not a very good database. It also ran counter to the idea that there should be no contact with S3 if there was a 100% disk cache hit rate. I also wanted horizontal scalability where many Rudolfs servers could run with their own separate disk cache and not run into any correctness issues.

Anyway, upstreaming this would be great. This would also reduce your maintenance burden going forward as it's a rather significant amount of changes compared to upstream. Happy to help shepherd PRs. Just please keep them small enough for easy review. :)

[rudoi]

Sounds good to me! I can't say exactly when - need to chat with my team - but we'll get some PRs open in the near future.

To respond to each note (though I'm sure we'll get into the details more at PR time):

1. Auth - understood, I can think of a few ways we could make this work through GitHub's API. Good callout!
2. Dynamo locks - We also were on a path to using S3, but we stumbled upon https://github.com/delta-incubator/dynamodb-lock-rs in our research and it inspired us to approach it using Dynamo. We ended up not using that crate though as we found it didn't match LFS semantics. I can't speak for any alternative lock storage backends, but it's been great for us so far. We've been using it for our game project since last ~August and our Dynamo usage has yet to put us past the free tier haha.

We've also implemented some things not included in the LFS locking spec, like batch locking endpoints. There are scenarios where Unreal needs to edit dozens or even hundreds of files at a time and our lock operations were totally destroying our game designers' workflows. We are totally fine maintaining our own repo for the purposes of possibly-Unreal-Editor-specific shenanigans.

Anyway, thanks for the reply!