Skip to content

Commit

Permalink
Update content/news/2024-11-04-biohackathon/index.md
Browse files Browse the repository at this point in the history
Co-authored-by: Beatriz Serrano-Solano <beatrizserrano@users.noreply.github.com>
  • Loading branch information
paulzierep and beatrizserrano authored Dec 12, 2024
1 parent 7142d60 commit 592185d
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion content/news/2024-11-04-biohackathon/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ Additionally, we explored ways to enable shared caching among users who opt in.
Galaxy’s Total Perspective Vortex (TPV) plugin offers powerful scheduling capabilities by routing jobs to destinations based on custom rules. At the BioHackathon, we expanded TPV's functionality to prioritize sustainability. We gathered detailed job statistics from Galaxy's database and compute nodes, but also from the remote job execution endpoints called [Pulsar](https://github.com/galaxyproject/pulsar). Because Pulsar endpoints don't require any port to be open to the public, we needed to find a way where Pulsar could send this information actively. This was achieved by a script on the Pulsar endpoint that consumes the queue's status and other information and sends it to a message queue ([RabbitMQ](https://www.rabbitmq.com/)) from which another script picks it up and sends it to our InfluxDB server. We developed ranking algorithms that direct jobs to the most environmentally friendly destinations. These algorithms take into account factors such as energy efficiency and resource usage to ensure that Galaxy workflows have a smaller carbon footprint.
### Network Boot Skill Sharing
Galaxy Europe's workload fluctuates, with idling servers during holidays and heavy job queues in peak seasons.
We are currently using an OpenStack instance where we spin up our images, so called 'Virtual Galaxy Compute Nodes' ([VGCN](https://github.com/usegalaxy-eu/vgcn)) which contain everything needed to pick up jobs from [HTCondor](https://htcondor.org/) and start crunching numbers.
We are currently using an OpenStack instance where we spin up our images, so-called 'Virtual Galaxy Compute Nodes' ([VGCN](https://github.com/usegalaxy-eu/vgcn)) which contain everything needed to pick up jobs from [HTCondor](https://htcondor.org/) and start crunching numbers.
Although the cloud is convenient for splitting large machines into smaller units, such as dedicated training infrastructure ([TiaaS](https://usegalaxy-eu.github.io/tiaas.html)), it is challenging to maintain, complex, and inflexible for power-saving. While we can stop cloud VMs, the underlying hypervisor hardware cannot be powered off.
Additionally, the hypervisor uses 5% of resources, and each VM uses another 5% for job management, leading to increased overhead, hardware wear, and energy use.
Paradoxically, simplifying and saving power requires some initial added complexity.
Expand Down

0 comments on commit 592185d

Please sign in to comment.