-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Small change to derive_predictors_and_scores
for speed + normalization
#119
base: main
Are you sure you want to change the base?
Conversation
…sResearch/collab-creatures into pp-collab2-pairwise-copying-pred
…ollab-creatures into pp-collab2-pairwise-copying-pred
…sResearch/collab-creatures into pp-collab2-howfarscore
Added option to scale values based on empirical CDF |
…uff (collab2 ). (#126) * Some work on the random_foragers notebook and fixing stuff. * Linting + completing the random_foragers notebook. * Finished random_foragers * Interactive plots now should be displayed in HTML * make format * small fixes to random foragers * Some more tweaks + zero-index fixes. * Hungry birds simulation updated. * Minor. * Completed the follower NB. * Saves the samples from each one of the R,H,F to disk, for later plotting in a single figure. * Comparative fig. * Minor. * Make lint and format * Typos * Improved explanations of the predictors and the scores. * Updated the model description in the random notebook. * Minor * Added option for initial positions. Updated RHF. * reviewed random * added toc to followers * fixed followers * Small formulas + model updates * small modification * small fixes, re-ran * fixing save and display in follower * format lint, dilling in hungry --------- Co-authored-by: rfl-urbaniak <rfl.urbaniak@gmail.com>
…/collab-creatures into pp-collab2-upgrade-derive
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please pull origin from the current version of staging, make sure you resolve all conflicts and pass all the tests.
derive_predictors_and_scores
for speedderive_predictors_and_scores
for speed + normalization
@rfl-urbaniak @dimkab Just finished going over this branch. The following changes were implemented:
Note: After merging these changes we would need to run all the doc notebooks again, as 2,3, and 4 can potentially change the outputs of the cells. I did not do this just yet so as to not overwhelm you both with 100 file changes in one PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not convinced that replacing merge with concat has no strange consequences at least for some of the predictors. As a sanity check I re-run communicators inference with your proposed modification and the posterior marginals are significantly different from the ones we currently have. I think this modification, if indeed correct, needs additional explicit tests that ensure proper functioning, involving predictors other than velocity too.
I noticed that generating a combined DF of predictors and scores was taking very long for large datasets (my kernel kept dying). This operation was very compute expensive because we were using
df.merge
which involves searching through column values to find matching rows. However, we can get away with using something simpler likepd.concat
since all predictor/score DataFrames inherit the grid fromlocal_windows
and hence the rows match by design. This should help clear the speed bottleneck.I also updated the function to return eCDF normalized values as an option