Integration of new API endpoints (V2)

[andregasser]

Addition of Mastodon API V2 endpoints

Hello everyone,

During the last couple of days I was trying to find a way on how to add the new Mastodon V2 API endpoints to this library in a good way. The high-level plan is to build a library which supports the current and the previous Mastodon API versions (currently V1 and V2).

I have collected some thoughs on this and would be very happy to get some feedback from you. It is very well possible that some of my thoughts below do not make sense at all. So please correct me 😀 Happy to get some input from experienced developers.

Mastodon Server API

• The number of API versions a Mastodon server exposes dependes on its version. A pre-4.0.0 Mastodon server exposes a V1 API, whereas a Mastodon server running 4.0.0 or higher exposes a V1 and a V2 API.
• Mastodon server API versions differ (was to be expected of course). An example is the call to fetch the instance information. Fields that have benn there in V1 are gone in V2, data types changes from simple Strings to JSON objects, ... Also it is to be expected, that requests for updating data will change or have changed already.

Mastodon4J Library

The current library targets V1 endpoints only. Also, not all of the V1 endpoints return all data possible. Some things are still missing. For example, in the call for fetching instance information a lot of data that is actually present, is not returned by the library.

Current Ideas

Idea 1: Library performs all the magic

The overall goal of this approach is to transform the different data models of Mastodon API V1 and V2 into some sort of unified data model that is presented to the library user. The user would not have to worry about which API version to call, as the library is doing this automatically (some sort of auto detection).

The unified data model would include:

• fields that are present in Mastodon API V1 only
• fields that are present in Mastodon API V2 only
• fields that are present in both Mastodon API versions

Furthermore, when a library user wans to post a status for example, he would not need to worry about which API version he is targeting. That means a "post status" request for example would:

• contain all the info that is needed to post a status towards Mastodon API V1
• contain all the info that is needed to post a status towards Mastodon API V2

Pros:

• Easier for the user
• In case of a new Mastodon API release, changes on library user side are reduced to updating changes from the library model.

Cons:

• A new unified data model needs to be introduced
• In general, more complexity on library side
• There might be cases that cannot be handled using this approach
• User would need to still perform version checks in order to find out, whether a field in the unified model is populated or not (remember, there are fields which are populated in a V1 call only, same for V2 calls).

Idea 2: Let the user choose

The goal of this approach is to support the current and the previous Mastodon API versions in the library (at the moment, V1 and V2). Code for every API version would evolve in completely separate packages(v1.* and v2.*). The user would have to choose for himself, which API client to use (V1 or V2). To make things easier, an additional helper class could be added to detect the Mastodon API version.

In case of a new Mastodon API version, a new v3.* package structure would be introduced. Also, library users need to migrate their code slowly over to the new v3.* code. v1.* package structure would be removed from library.

Pros:

• Library maintenance seems much easier
• Client implementations for Mastodon API versions are cleanly separated and can evolve independently

Cons:

• Code duplication due to v1.* and v2.* packages

Personal Opinion

At the moment I am favoring idea number 2, as it seems much more easier to maintain. The additional burden on the user side to decide whether to use V1 or V2 client seems to be acceptable. I think the approach in idea 1 is calling for trouble and complex scenarios. I rather have a client for every API version that returns a proper representation of the data that is currently there.

[bocops]

For what it is worth, only few V2 endpoints actually exist. It is hard to tell from the web documentation alone, so I dug up routes.rb, which has roughly 240 lines of code for V1 endpoints, vs. about 20 for V2 endpoints. Another idea is to look at /app/controllers/api for endpoints that each version implements - where, again, the V1 subdirectory is much larger than V2.

What that means is that a library user using some V2 endpoints couldn't use just the "V2 client", but would also need to use the separate(?) V1 client for the ~90% endpoints that don't yet have a V2 counterpart.

In some of these cases, V1 and V2 endpoints seem to be at least compatible with each other, both as far as the necessary information to build either a V1 or a V2 query is concerned, and also concerning the returned data. For example, both versions of /admin/accounts (doc link) return the same "Array of Admin::Account" and only differ in some compatible query options (for example origin string vs. individual local and remote booleans). In fact, there are currently only two entities that are marked as "V1" (Filter and Instance), so supposedly only these have changed in an incompatible way.

With all of that in mind, what about this:

Idea 3: Mirror what Mastodon does, and use Semantic Versioning to notify library users without doing too much "magic"

• if V1 and V2 endpoints are compatible as described above:
  • start using V2 endpoint internally but continue offering a method with signature matching the previously defined V1, which does the mapping and hides implementation details from library users. Increase minor version number of this library (e.g. 1.7.0 -> 1.8.0) when releasing this addition.
  • Eventually mark as deprecated the old method signature when releasing something with an increased major version number (e.g. 1.x.y -> 2.0.0).
  • Later, remove deprecated method in another major update (2.x.y -> 3.0.0)
• if V1 and V2 endpoints are not compatible, do the following:
  • First, rename existing method to something else, then add a new method with the previous name for V2 functionality. Do the same for classes (such as Instance -> V1Instance). Increase major when releasing this change. Library users at this point can update their code to continue using V1 functionality.
  • Later, deprecate the renamed methods and classes when Mastodon deprecates their matching V1 endpoint, in a minor release.
  • Much later still, remove renamed methods and classes if sensible at that point, in a major release.

This leaves the complication of the same version of this library potentially having to deal with different Mastodon server versions - but I think between

• sensible fallbacks in individual methods (use V1 if V2 is unsupported),
• default return values (return null or empty lists if a V2-only method is called on a V1 server) and
• making the server version available to library users, so that they can put some guard clauses in their code
  this should be manageable.

[andregasser]

Hello @bocops. I was not aware that the V2 API is "not really there yet". Considering this, your approach makes sense 😀 Let's go that route. Cheer's André

[bocops]

I was thinking about this some more, and am currently wondering whether we need to lay some foundations for supporting instances on different server versions for our release 2.0.0, or if that can wait until later.

I'm using View all followed tags and File a report as examples, because they have multiple changes in the version range that is relevant to us (current and last major, so anything >= 3.0.0).

In our code to view all followed tags, we might want to do something along these lines:

if (instance.version >= "4.1.0") {
    // supports pagination headers, so use them
} else if (instance.version >= "4.0.0") {
    // no header support, so don't use them
} else {
    // endpoint is unsupported, so we need to fail somehow
}

Similarly, when filing a report:

if (instance.version >= "4.0.0") {
    // file report when at least `category` OR `rule_ids` is given; else fail(?)
} else if (instance.version >= "3.5.0") {
    // file report when `category` AND `rule_ids` are given; else fail(?)
} else {
    // endpoint is supported since 2.3.0, which means that all server versions we want to support have it; file report whether or not parameters are given
}

Based on that, some follow-up questions:

1. We currently don't have access to the instance version in our methods. MastodonClient is built with just the instance hostname as a string. We would need to perform a call to Public.getInstance() first, then store that instance or at least its version field with our client somehow. Should this be done automatically, or do we want to let the user beware and only check version if it has been made available to us be the user?
2. If we fail, how do we want to fail? With an exception (whether that is BigBoneException or a new subclass thereof), silently where possible, or something else?
3. If failure based on server version is an option, do we need functionality that allows library users to check whether they can perform a certain action before they actually attempt doing so? If so, what would be the best way to implement this?

[andregasser]

Hello @bocops I think we need to do something like you describe. I have tried to summarize the outcome of this discussion here in a nice and clean wiki page here: https://github.com/andregasser/bigbone/wiki/API-Support. Happy to get some feedback from you. Feel free to add your comments (either here or directly in the wiki page).

[bocops]

Wow, looks great at a first glance! I will have a more thorough look later, but so far I can't see any problems.

Regarding the actual implementation, unless you are already working on it I can try and suggest something for the version detection in MastodonClient and general version comparison in a PR.

[andregasser]

I have implemented the automatic version detection for MastodonClient already - it is laying around in some branch. I suggest I'll re-add it to this project in a sep. feature branch (hopefully this evening).

[bocops]

I'm copying parts a comment from #94 here, we should discuss whether we want a structure like that for our entities that come in V1 and V2 variants:

For what it's worth, if we want to store either the V1 or V2 instance in our MastodonClient, then we would probably have to have an (abstract?) Instance super class containing common fields between the V1 and V2 responses, plus separate InstanceV1 and InstanceV2 sub-classes that each have fields unique to that version. Then, when accessing one of the api/vX/instance endpoints, we would then deserialize into the matching versioned instance class, but have a member of the super class in MastodonClient. In turn, library users would then have to use is/instanceOf when working with that member.

[bocops]

Supporting different Mastodon server versions / API levels was brought up in #217 again. With some more months of both using Mastodon and implementing its APIs, at least my understanding of the whole situation has changed a bit:

Mastodon version 3.0.0 was released in October 2019, nearly four years ago. Visiting different instances in the past months, I haven't found a single one that is running anything less than 4.0.0, let alone 3.0.0. If we keep our promise to support everything down to 3.0.0, we might find ourselves spending a lot of time on overcomplicating our codebase for something that won't ever be used. So, one thing we could consider would be to restrict our compatibility to "minor Mastodon server versions first released within the last X months, restricted to the current and last major version".

For what it's worth, if X=12, which seems to be a reasonable timeframe for active Mastodon instances to be updated, then the final minor release of major version 3 (3.5.0) would already be older than that and (if we ignored backports to older versions) we could just focus on version 4 for the time being.

Even picking a larger X=24 would mean support for either 3.5.x or at most 3.4.x (first released in May 2021), but not everything down to 3.0.0.

Last but not least, even if we want to keep supporting "the current and last version down to their first releases", we could choose to start this with the current version - so, keep supporting 4.x once 5.0 comes around, but not necessarily get everything in place for 3.x.

That aside, I believe most of the other ideas in this discussion are still valid. Once we support different server versions with different capabilities that are not strictly backwards-compatible, we need to decide whether we want to make information about that available to library users, or whether it is sufficient to potentially throw a RequestException and let the library user deal with that.