[Bug]: dstack Sky UI Does Not Reflect Correct Status After Scaling Up/Down

[movchan74]

Steps to reproduce

1. Start a run with one replica using dstack apply:

   dstack apply

2. Scale the run to two replicas using the scaling script:

   python scale_run.py --project <project> --run-name <run-name> --token <token> --replicas 2

3. Observe the following:

   • The run scales to two replicas.
   • The logs in the UI are not displayed for either of the replicas (No logs to display.).
   • dstack stats confirms two replicas are running.
   • Run price doubles as expected.

4. Scale the run back to one replica using the scaling script:

   python scale_run.py --project <project> --run-name <run-name> --token <token> --replicas 1

5. Observe the following:

   • The run scales back to one replica.
   • The logs in the UI are still missing (No logs to display.).
   • dstack stats still shows two replicas running, though only one should remain.
   • The run price remains doubled, though it should be reduced.

6. Scale the run back to two replicas again:

   python scale_run.py --project <project> --run-name <run-name> --token <token> --replicas 2

7. Observe the following:

   • The run scales to two replicas.
   • The logs in the UI remain missing (No logs to display.).
   • dstack stats shows two replicas running.
   • The run price doubles as expected.
   • dstack Sky UI incorrectly shows the second replica as Terminated, while dstack ps correctly displays its status as running.

8. Use stop button on the run page and observe that it doesn't do anything.

9. Go to runs list, choose run and use stop button. Observe that it works.

Actual behaviour

• Scaling up and down works functionally, but the UI does not reflect the correct state.
• Logs are not displayed in the UI after scaling.
• dstack stats does not update correctly when scaling down to one replica (it still shows two).
• Run price remains higher than expected after scaling down.
• dstack Sky UI incorrectly marks a running replica as Terminated when scaling back up.
• Stop button on the run page doesn't work.
• Stop button on the list of runs works correctly.

Expected behaviour

• The UI should display logs for all running replicas.
• dstack stats should correctly reflect the actual number of running replicas after scaling down.
• Run price should adjust accordingly after reducing the number of replicas.
• dstack Sky UI should correctly display the status of all replicas.
• Stop buttons should work everywhere.

dstack version

0.18.42

Server logs

Additional information

• The issue persists even after refreshing the UI.
• dstack ps correctly shows the replica status, but the Sky UI does not match.
• The scaling script executes successfully, meaning the backend scaling process works—this appears to be a UI sync issue.
• Scaling script is available here: https://gist.github.com/movchan74/85a6157ab3274afbdcb08f4513be6a36

[jvstme]

@movchan74, thanks for reporting these bugs. You can track the progress of the fixes in sub-issues.