How to use other protobufs?

[johnpyp]

As noted in the readme, this project should be able to work with other protobufs not generated from steam-vent... it would be great to have an example :) E.g I don't know if I'm supposed to wrap my own protobuf-encoded body with something like a CMsgProtobufWrapped... it could make sense, but I'm not sure. My protobufs are currently generated with Prost.

[icewind1991]

You have to use the steam-vent-proto-build from protobuf/build to generate the protobuf structs.

steam-vent-proto-build path/to/protos path/to/output/sources

The output structs from that should work directly without any further work.

[johnpyp]

I see, thanks! I cloned the repo locally and generated deadlock and it all seems to work... the one thing I'm still stuck on is which of these generated types make NetMessage's to send to the client. It seems there are a bunch of Messages that don't implement NetMessage, but they're commands that are meant to be sent to the GC. Is there some generic wrapper type I'm supposed to use? I'm using .job(...), is there some other method that's specific to these kind of types maybe?

For example CMsgClientToGCSpectateLobby - https://github.com/SteamDatabase/Protobufs/blob/57781c1d10d35910fe3d1918b3d1e429e6488c37/deadlock/citadel_gcmessages_client.proto#L779

This example using node-steam-user just encodes the type and binary serialization separately... https://github.com/DanielAtanasovski/deadlock-history/blob/61f3496398d534ad850d96ece1207b79dd9f1fb0/src/matches.ts#L47-L53 Is there something comparable we're supposed to do here?

[icewind1991]

For GC messages you can create a GameCoordinator from the connection (see the backpack example).

[johnpyp]

Right, I'm modelling it after the backpack example... however in the backpack example, the messages being sent still implement NetMessage. Here's my code I'm trying:

use steam_vent::{
    auth::{
        AuthConfirmationHandler as _, ConsoleAuthConfirmationHandler, DeviceConfirmationHandler,
        FileGuardDataStore,
    },
    proto::deadlock::citadel_gcmessages_client::CMsgClientToGCSpectateLobby,
    Connection, ConnectionTrait, GameCoordinator, ServerList,
};

// pub struct

pub async fn run() -> anyhow::Result<()> {
    let username = std::env::var("STEAM_USERNAME").expect("STEAM_USERNAME env var must be set");
    let password = std::env::var("STEAM_PASSWORD").expect("STEAM_PASSWORD env var must be set");

    let server_list = ServerList::discover().await?;
    let connection = Connection::login(
        &server_list,
        &username,
        &password,
        FileGuardDataStore::user_cache(),
        ConsoleAuthConfirmationHandler::default().or(DeviceConfirmationHandler),
    )
    .await?;

    println!("starting game coordinator");

    let game_coordinator = GameCoordinator::new(&connection, 1422450).await?;

    let spectate_message = CMsgClientToGCSpectateLobby {
        lobby_id: Some(101038497616221460),
        ..Default::default()
    };

    let res = game_coordinator.job(spectate_message).await?;

    Ok(())
}

pub fn run_sync() -> anyhow::Result<()> {
    tokio::runtime::Builder::new_current_thread()
        .enable_all()
        .build()
        .unwrap()
        .block_on(run())
}

and the error message:

[icewind1991]

Looks like the logic for matching the protobuf structs to the type/kind isn't working for (all of) the deadlock structs.

I'll see about better supporting messages that don't have the kind mapped. You can manually set the mapping by implementing RpcMessageWithKind for the struct in the meantime.

[johnpyp]

Ah ok, makes sense, thanks!

[johnpyp]

So, implementing this workaround manually in the crate was pretty easy:

mod generated;

pub use generated::*;

use steam_vent_proto_common::{MsgKindEnum, RpcMessageWithKind};

use crate::citadel_gcmessages_client::EGCCitadelClientMessages;

impl MsgKindEnum for EGCCitadelClientMessages {}

impl RpcMessageWithKind for generated::citadel_gcmessages_client::CMsgClientToGCSpectateLobby {
    type KindEnum = EGCCitadelClientMessages;

    const KIND: Self::KindEnum = EGCCitadelClientMessages::k_EMsgClientToGCSpectateLobby;
}

impl RpcMessageWithKind
    for generated::citadel_gcmessages_client::CMsgClientToGCSpectateLobbyResponse
{
    type KindEnum = EGCCitadelClientMessages;

    const KIND: Self::KindEnum = EGCCitadelClientMessages::k_EMsgClientToGCSpectateLobbyResponse;
}

However, before that I tried to implement it using the NewType pattern in my crate because these are all foreign types... which was basically impossible given all the traits needed to be implement.

Just for the future, it might be worth implementing some "bailout" methods that can just accept the raw stuff needed, kind of like the node-steam-user example. As of right now, I think it would be near impossible for a user to use their own custom protobufs that happen to not work quite right with this custom proto codegen.

[icewind1991]

Thanks for the feedback

[johnpyp]

@icewind1991 I managed to get the RpcMessageWithKind generating for all of the messages I was expecting, with this naive approach:

diff --git a/protobuf/build/src/main.rs b/protobuf/build/src/main.rs
index 22df538..00fa3c4 100644
--- a/protobuf/build/src/main.rs
+++ b/protobuf/build/src/main.rs
@@ -338,8 +338,10 @@ impl CustomizeCallback for ServiceGenerator {
         let kind_enums: Vec<_> = file
             .enums()
             .filter_map(|enum_type| {
-                (enum_type.name().starts_with("E") && enum_type.name().ends_with("Msg"))
-                    .then(|| enum_type.name().to_string())
+                (enum_type.name().starts_with("E")
+                    && (enum_type.name().ends_with("Msg")
+                        || enum_type.name().ends_with("Messages")))
+                .then(|| enum_type.name().to_string())
             })
             .collect();
 
@@ -423,7 +425,10 @@ fn get_kinds(base: &Path, protos: &[PathBuf]) -> Vec<Kind> {
                 .iter_mut()
                 .map(move |e| (mod_name.clone(), e))
         })
-        .filter(|(_, e)| e.name().starts_with("E") && e.name().ends_with("Msg"));
+        .filter(|(_, e)| {
+            e.name().starts_with("E")
+                && (e.name().ends_with("Msg") || e.name().ends_with("Messages"))
+        });
 
     let mut kinds = kinds_enums
         .flat_map(|(mod_name, kinds_enum)| {
@@ -436,14 +441,17 @@ fn get_kinds(base: &Path, protos: &[PathBuf]) -> Vec<Kind> {
             let prefix = prefix[0..prefix.len() - 1].to_string();
             let variant_prefix = format!("k_EMsg{}", prefix);
             let variant_prefix_alt = format!("k_E{}Msg_", prefix);
+            let variant_prefix_alt2 = "k_EMsg".to_string();
             let enum_prefix = prefix.to_ascii_lowercase();
             let enum_name = kinds_enum.take_name();
+
             kinds_enum.value.iter_mut().map(move |opt| Kind {
                 mod_name: mod_name.clone(),
                 enum_name: enum_name.clone(),
                 enum_prefix: enum_prefix.clone(),
                 variant_prefix: variant_prefix.clone(),
                 variant_prefix_alt: variant_prefix_alt.clone(),
+                variant_prefix_alt2: variant_prefix_alt2.clone(),
                 variant: opt.take_name(),
                 struct_name_prefix_alt_len: prefix.len(),
             })
@@ -462,6 +470,7 @@ struct Kind {
     enum_prefix: String,
     variant_prefix: String,
     variant_prefix_alt: String,
+    variant_prefix_alt2: String,
     variant: String,
     struct_name_prefix_alt_len: usize,
 }
@@ -474,6 +483,7 @@ impl Kind {
             .variant
             .strip_prefix(&self.variant_prefix)
             .or_else(|| self.variant.strip_prefix(&self.variant_prefix_alt))
+            .or_else(|| self.variant.strip_prefix(&self.variant_prefix_alt2))
         else {
             return false;
         };

The two main issues are:

1. Enum ending in Messages instead of Msg
2. k_EMsgClientToGC... messages (e.g CMsgClientToGCSpectateLobby), not following the existing prefix scheme at all (i.e no "prefix" to strip in the first place).

I hope this helps fix in a more robust way 🙏

---

I'm not sure if it's related, but I haven't been able to use .job with any of these messages either, despite getting the Response back clearly in the debug tracing logs. E.g

        let res = game_coordinator
            .job::<CMsgClientToGCSpectateLobby, CMsgClientToGCSpectateLobbyResponse>(
                spectate_message,
            )
            .await?;

always times out, despite the fact I see both the SpectateLobby request get sent out, and a message with the same kind as SpectateLobbyResponse come back in.

[icewind1991]

3fa7376 allows sending types implementing EncodableMessage (a trait with just the encode/decode parts from NetMessage) with raw_send_with_kind. Still need to come up with a good way to do the same for receive.

[icewind1991]

Merged the protobuf builder changes with 2874fce thanks for the patch.

[icewind1991]

I'm not sure if it's related, but I haven't been able to use .job with any of these messages either, despite getting the Response back clearly in the debug tracing logs. E.g

        let res = game_coordinator
            .job::<CMsgClientToGCSpectateLobby, CMsgClientToGCSpectateLobbyResponse>(
                spectate_message,
            )
            .await?;

always times out, despite the fact I see both the SpectateLobby request get sent out, and a message with the same kind as SpectateLobbyResponse come back in.

Does the response have the job id set correctly?

Can you show the relevant log lines from running with RUST_LOG=debug

[johnpyp]

I tracked down the issue - https://github.com/icewind1991/steam-vent/blob/main/src/connection/mod.rs#L397

Specifically self.filter().on_job_id(header.source_job_id); should actually be self.filter().on_job_id(header.target_job_id);.

Adding logging there, I noticed the source job id was like 1 or 2 sometimes, where the target job id was the "real" one I was expecting. When I switched it, I started getting responses back as expected.

EDIT: I think it's something else actually? This seems to just be working coincidentally as its the hardcoded NONE job id... when there are no other messages potentially interfering concurrently it works, but I imagine it'll break easily.

[johnpyp]

Ok, I think I actually identified the issue - it's this function: https://github.com/icewind1991/steam-vent/blob/main/src/game_coordinator.rs#L165-L188

Specifically, the nested_header business, which kind of "erases" the job id from the perspective of the server... If I just replace

nested_header.write(&mut payload, kind, is_protobuf)?;

with

header.write(&mut payload, kind, is_protobuf)?;

then it all seems to work. The job id gets properly sent, and the server responds in kind.

This is also consistent with the behavior of node-steam-user afaict, which is what I've been comparing against to debug this.

[icewind1991]

Can you try it with

msg.process_header(&mut nested_header);

before the nested_header.write

[johnpyp]

That doesn't seem to change anything in this case