Wondering if we can add some warnings on data transfers

[codexetreme]

I recently created a K3s cluster with a 3 node setup on AWS ASGs. And an external postgres DB via cockroachDB. Rest all things are default.

Over the next 3 days it exchanged about 5tb of data between the nodes over the global internet. And on cockroach DB, my instance (which is a free tier unlimited scale up cluster, these are the defaults on the cockroach DB side) , it used over 6 billion RUs (read units) Leading to a bill of 1.3K USD (this might not be massive for large orgs, but for startups and hobbyists its a problem)

I was wondering about the following things

• If this is used in an IOT usecase the gossip will eat up valuable bandwidth on the comm bus
• On cloud, we should probably add some more information about the huge amount of R/Ws on external databases (I understand its meant to work like etcd and needs to talk to the DB constantly, but we need to warn users of this)
• Add some more documentation where it show how to send the gossip protocol data between the nodes via private IPs in a cloud environment (I still havent figured out how to do this, but I am guessing it involves tinkering with either flannel or the wireguard protocol options)

What are my specific questions ?

• How to setup gossip comms over private IPs in a cloud setup
• Can we add a warning about excessive data R/W requirements on an external Datastore
• Can we have some prometheus style metrics on the cluster's networking, datastore networking ,etc etc (if there are already, then I definitely missed these)

[brandond]

None of these things are really specific to K3s.

1. K3s does not use any sort of gossip networking?
2. The K3s datastore requirements shouldn't be excessive - they vary directly in proportion to reads and writes against the Kubernetes apiserver. Note that by default K3s uses embedded sqlite or etcd; if you chose to point it at a managed database service, you should definitely be prepared to monitor costs for that service.
3. There are prometheus metrics available for everything that comes with k3s. You'd need to deploy a monitoring stack to scrape and visualize them though; K3s does not come with prometheus/grafana or anything along those lines.