Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add visualization tool and alibaba_loader_task_max_pow2_slots flag and more #79

Merged
merged 50 commits into from
Dec 19, 2023
Merged

Add visualization tool and alibaba_loader_task_max_pow2_slots flag and more #79

merged 50 commits into from
Dec 19, 2023

Conversation

ruizehung
Copy link
Contributor

@ruizehung ruizehung commented Dec 19, 2023 •
edited
Loading

  • Add a streamlit app to visualize individual run
  • Add a new flag --alibaba_loader_task_max_pow2_slots to main.py
  • Enable Critical Path optimization pass in schedulers/tetrisched/src/OptimizationPasses.cpp
  • Use round instead of int in fuzz_time
  • Fix POISSON arrival time generation. Previously we pass in arrival rate into self._rng.poisson, which actually wants "inter-task" arrival rate.
  • Clip the min deadline of tasks to at least 5 time unit to prevent some small taks from having tight deadline.
  • Refactor data/csv_types.py and data/csv_reader.py to use data class
    • Now we also log tasks that get cancelled but didn't get released
  • Some updates to experiments/analysis_utils.py

ruizehung and others added 30 commits December 7, 2023 22:30
@ruizehung
Fix TETRISCHED_LOGGING_DIR_ENV_NAME in timing::ScopeTimer
0646183
@ruizehung
Enable CriticalPathOptimizationPass
43f7c53
@ruizehung
Add 12/7 experiment scripts
067e510
@ruizehung
Have TASK_GRAPH_RELEASE logged to info
7c12756
@ruizehung
Add 12/8 experiment scripts
44c3e1b
@ruizehung
- Parse dynamic and adpative discretization in extract_variables_from…
caef13b 
…_filename_v2

- Skip reading libtetrisched_performance.csv in analysis_utils.py
@ruizehung
Fix TASK_GRAPH_RELEASE logging in __handle_task_graph_release
7c499d7
@ruizehung
Fix some type hint and not add canceled tasks that weren't released
99b86c0
@ruizehung
Fix analyze_task_placement
40e0584
@ruizehung
Update 12/8 experiment script
5eb176b
@ruizehung
Add analyze_resource_utilization_by_arrival_rate_and_cv2
1e8bfab
@ruizehung
Disable tetrished performance logging
e2a973b
Added straight chain dag pkl file and expt script for 9th dec runs
11bdf82
Updated straight chain dag expt for quicker run
2d891c7
Removed few erroneous straight chain dags
bcbd927
@ruizehung
Add Promtail
70ba62c
Adding 12/9 expts and pkl with min len straight chain dags of 4
843fd6f
Minor fix in 12/9 expt script
4aef40a
Updates to 12/9 expt script
d1210e9
Second night run, 12/9, task_cpu_div=10
5ab3374
@ruizehung
Merge branch 'ray-osdi-experiment' of https://github.com/erdos-projec…
eaabdf8 
…t/erdos-scheduling-simulator into ray-osdi-experiment
@ruizehung
Ignore "input_flag" in CSVReader
345154a
@ruizehung
Add 12/9 experiments scripts
8672486
@ruizehung
Revert "Add Promtail"
01e98fa 
This reverts commit 70ba62c.
@ruizehung
Make task and task_graph in csv_types as dataclass so that we can loa…
0dac660 
…d them to pandas
@ruizehung
Add individual run analysis notebook
d95985a
@ruizehung
Add a pkl file that has 6100 jobs with critical path >= 4
0f1af15
@ruizehung
Add experiment script for 12/10 night
a88065a
@ruizehung
Add window_to_execute field to TaskGraph in csv types
0f08d7f
@ruizehung
Fix poisson policy inter_arrival_times generation
988b718
Ray Hung and others added 18 commits December 15, 2023 10:37
Checkpointing simon changes, includes num slots override in powers of…
277a37d 
… two, fix to minimum task deadline
@ruizehung
Use round in fuzz time instead of int
0ba53fe
@ruizehung
prevent CSVReader from outputing unknown event "TASK_SKIP"
12e0d9e
@ruizehung
fix and update analysis_utils.py
139c8f1
@ruizehung
Remove jobs_having_ctp_gt_4.pkl
e07f5a4
@ruizehung
- Remove task_duration_multipler
a0fbdf3 
- Add _sample_normal_distribution_random() to randomize task duration
@ruizehung
Merge branch 'ray-osdi-experiment' of https://github.com/erdos-projec…
a8d12e6 
…t/erdos-scheduling-simulator into ray-osdi-experiment
@ruizehung
add comment for middle heavy task duration code
3aa8e0e
@ruizehung
Merge branch 'main' into ray-osdi-experiment
4815f46
@ruizehung
Add 12/12 night experiment script
cc52c92
@ruizehung
Add 12/15 result
64176fc
@ruizehung
Merge branch 'main' into ray-osdi-experiment
0adce53
@ruizehung
Log slowest task execution time in task cancel
8ef2959
@ruizehung
- Also log task that got cancelled without being released first in cs…
84445cb 
…v logs

- Convert Resource, WorkerPoolUtilization, and WorkerPool to dataclass
- Add slowest_execution_time field to Task
- Make intended_release_time, release_time, deadline, and window_to_execute optionally None
- Add Task.release_time_compare_key to support comparing with None values
@ruizehung
Convert WorkerPoolStats to data class and add Resource.__repr__
3dbf5a0
@ruizehung
Add Streamlit visualization for individaul runs
7912c65
@ruizehung
Update streamlit to have separate tables for meet deadline, cancel, a…
81f7da2 
…nd miss deadline tasks and tasks graphs
@ruizehung
Temporarily disable random task duration to merge into main
f6c891d
@ruizehung ruizehung changed the title Add visualization tool and alibaba_loader_task_max_pow2_slots flag Add visualization tool and alibaba_loader_task_max_pow2_slots flag and more Dec 19, 2023
@ruizehung ruizehung merged commit 3a47288 into main Dec 19, 2023
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@ruizehung