Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

5001-node-selection-algorithm speed increase. #27

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

burdettadam
Copy link

@burdettadam burdettadam commented Dec 6, 2019

20 minutes to less than 10 seconds on my test data.
To get this speed increase:

  • I removed unique_combinations and used combinations from itertools.
  • I utilized Pool from multiprocessing to do a MapReduce on batches from unique_combinations.
  • I removed dead code and reorganized some things to reduce calculations.
    I do not have time to work on this code anymore. If needed, more speed could be achieved by using panda to read in the config and create a table that would allow unique combinations to be represented by indexes of that table. this would cut down some un-needed computation. ComboAnalysis could also be optimized by having a pre-allocated array for maintaining the best combinations. I saw significant time being spent in an array append method inside ComboAnalysis, with the way batches are being processed, this would be a significant speed increase.
    Another possible speed increase would be using a library like numpy for all math operations.
    This code needs to be tested more than what I have done.

Lynn Bendixsen and others added 2 commits November 26, 2019 10:45
Signed-off-by: Lynn Bendixsen <lynn@sovrin.org>
Signed-off-by: Adam Burdett <burdettadam@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant