[Feat] Add Local Search for Solution Improvement #140

hyeok9855 · 2024-03-15T15:42:39Z

Description

Add local search operators as a post-processing to improve a given solution.
Here, we implement 2-opt for TSP and LocalSearch operator provided by PyVRP for CVRP.
For other problems (e.g., PDP or scheduling), we couldn't find such a plug-and-play local search operator, and we're looking for contributions to local search for other problems!

Note that I also made some refactorings regarding type hinting.

Motivation and Context

Local search is an essential component for an enhanced CO solver. Many research projects in the NCO field utilize local search operators, such as our recent work GFACS, which trains NN using the solution refined by local search in an off-policy manner.

I have raised an issue to propose this change (required for new features and bug fixes)

Types of changes

What types of changes does your code introduce? Remove all that do not apply:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds core functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation (update in the documentation)
Example (update in the folder of examples)

Checklist

Go over all the following points, and put an x in all the boxes that apply.
If you are unsure about any of these, don't hesitate to ask. We are here to help!

My change requires a change to the documentation.
I have updated the tests accordingly (required for a bug fix or a new feature).
I have updated the documentation accordingly.

rl4co/envs/common/base.py

rl4co/envs/routing/tsp.py

tests/test_envs.py

fedebotu · 2024-03-16T03:19:24Z

Great job! 🚀

Another detail I cannot review above: tt seems that the installation fails for Python=3.8. This should be because pyvrp is only available for Python 3.9 onwards (reference), so maybe we can skip its for that case

rl4co/envs/routing/tsp.py

pyproject.toml

rl4co/envs/routing/tsp.py

rl4co/envs/common/base.py

…ption to DeepACO

hyeok9855 · 2024-05-01T19:04:36Z

Changes

Merged the updated main branch
Added the reward augmentation with local search, which is one of the key components of DeepACO.

hyeok9855 · 2024-05-31T22:20:06Z

Changes

Merge the main branch (c.f., [BugFix] Fix the performance issue of DeepACO #170)
Increase the performance further (now it's very close to the performance of the original implementation of DeepACO)
Add CVRP local search based on PyVRP

@fedebotu @leonlan
If you have time, please review the renewed code!

rl4co/envs/routing/cvrp/local_search.py

leonlan · 2024-06-01T05:52:06Z

Hi @hyeok9855, I don't have time to go over the code in full detail, but I had a quick glance at the PyVRP part. It looks good to me with a small comment on the constant.

fedebotu · 2024-06-01T06:56:35Z

rl4co/models/zoo/deepaco/antsystem.py

@@ -27,7 +29,10 @@ class AntSystem:
        pheromone: Initial pheromone matrix. Defaults to `torch.ones_like(log_heuristic)`.
        require_logprobs: Whether to require the log probability of actions. Defaults to False.
        use_local_search: Whether to use local_search provided by the env. Default to False.
-        local_search_params: Arguments to be passed to the local_search function.
+        use_nls: Whether to use neural-guided local search provided by the env. Default to False.


What is the difference between use_nls and use_local_search?

nls indicates the local search with neural-guided perturbation, proposed in DeepACO. See here.

rl4co/envs/routing/tsp/env.py

fedebotu · 2024-06-01T07:04:04Z

rl4co/envs/routing/tsp/env.py

@@ -166,6 +167,19 @@ def check_solution_validity(td: TensorDict, actions: torch.Tensor):
            == actions.data.sort(1)[0]
        ).all(), "Invalid tour"

+    def generate_data(self, batch_size) -> TensorDict:


[Important] We replaced the generate_data with @cbhua this function with the more modular generator function:(e.g. here).

To generate data, we can call: env.generator(...) instead of env.generate_data

So you mean I can just remove the method?

fedebotu · 2024-06-01T07:06:46Z

rl4co/models/zoo/deepaco/antsystem.py

+        return heuristic_dist
+
+    @staticmethod
+    def select_start_node_fn(


[Minor, for now]
By default, now we look for this function inside of the environment as done here, which is a bit more modular. But since we have to transfer this functions yet and is not a hard task, no need to do it now

fedebotu · 2024-06-01T07:08:45Z

rl4co/models/zoo/deepaco/antsystem.py

-        return actions, reward  # type: ignore
+        td_cpu = td.detach().cpu()  # Convert to CPU in advance to minimize the overhead from device transfer
+        td_cpu["distances"] = get_distance_matrix(td_cpu["locs"])
+        # TODO: avoid or generalize this, e.g., pre-compute for local search in each env


Yes, we can keep this as todo, but it should be generalized. I think this could be a common classmethod for routing environments

fedebotu · 2024-06-01T07:12:22Z

rl4co/models/zoo/nargnn/encoder.py


-        return heatmaps_logits
+        heatmap += 1e-10 if heatmap.dtype != torch.float16 else 3e-8


Nice, I see a huge trick here!
These values should ideally be constants at the top, e.g:

LOWEST_POSVAL_FP32 = 1e-10 LOWEST_POSVAL_FP16 = 3e-8

The lowest positive value for FP32 is not 1e-10 actually. It is much smaller than that, but 1e-10 is used in DeepACO.

fedebotu · 2024-06-01T07:13:33Z

tests/test_envs.py

@@ -79,6 +79,10 @@ def test_routing(env_cls, batch_size=2, size=20):
 def test_mtvrp(variant, batch_size=2, size=20):
    env = MTVRPEnv(generator_params=dict(num_loc=size, variant_preset=variant))
    reward, td, actions = rollout(env, env.reset(batch_size=[batch_size]), random_policy)
+    try:
+        env.local_search(td, actions)


Did you add the local search to this environment?

We should have this available for all 16 variants with the code @leonlan made

No, I only added LS to TSP and CVRP.
I don't remember if I added the lines. This might be due to the unexpected behavior of git merge.
I will just remove it for now!

We should have this available for all 16 variants with the code @leonlan made

Where can I find the code? Is that about local search?

It's about full solutions, but my guess is that it will work with any variant with minor modifications! Here is the code~

fedebotu · 2024-06-01T07:16:55Z

I think we should skip the local search testing with pyvrp for Python < 3.9, I think that is why testing failed

Like:

@pytest.mark.skipif(sys.version_info < (3, 9))
[...]

Maybe in the near future we could just remove testing Python 3.8 since it's a really old version anyways

hyeok9855 · 2024-06-08T08:23:46Z

Unresolved comments will be dealt with in the following PR.

fedebotu reviewed Mar 16, 2024

View reviewed changes

rl4co/envs/common/base.py Outdated Show resolved Hide resolved

rl4co/envs/routing/tsp.py Outdated Show resolved Hide resolved

rl4co/envs/routing/tsp.py Outdated Show resolved Hide resolved

tests/test_envs.py Outdated Show resolved Hide resolved

leonlan reviewed Mar 16, 2024

View reviewed changes

rl4co/envs/routing/tsp.py Outdated Show resolved Hide resolved

rl4co/envs/routing/tsp.py Outdated Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

rl4co/envs/routing/tsp.py Outdated Show resolved Hide resolved

fedebotu mentioned this pull request Mar 18, 2024

[Feat] Add DeepACO #142

Merged

4 tasks

fedebotu reviewed Mar 18, 2024

View reviewed changes

rl4co/envs/common/base.py Outdated Show resolved Hide resolved

hyeok9855 added 4 commits April 26, 2024 21:28

correct wrong type hintings

b325dbb

Add 2-opt local search for TSPEnv

01f254c

Add tutorial notebook for solution improvement

12b702c

Replace pyvrp-based twoopt with custom one from DeepACO

7ac643d

hyeok9855 force-pushed the local-search branch from 95f24ea to 7ac643d Compare April 26, 2024 15:28

hyeok9855 added 3 commits April 27, 2024 02:28

Add neural-guided neural-guided perturbation and local search (NLS) o…

ccbcd9b

…ption to DeepACO

Merge branch 'main' into local-search

c91ca02

Implement local search based reward augmentation for DeepACO

6e91ecc

separate local search from env.py

77f2183

hyeok9855 force-pushed the local-search branch from c1c5637 to b6520cf Compare May 4, 2024 06:11

add DeepACO-style select_start_node_fn for multistart

d19e20c

hyeok9855 force-pushed the local-search branch from b6520cf to d19e20c Compare May 4, 2024 06:47

hyeok9855 mentioned this pull request May 17, 2024

[BugFix] Fix the performance issue of DeepACO #170

Merged

hyeok9855 added 6 commits May 28, 2024 20:56

Add manual advantage calculation

77defb7

Merge branch 'main' into local-search

a84e5e8

minor fix

6d65f15

resolve the performance drop

1b5f112

add pyvrp-based local search for CVRP

0bf80d5

minor refactoring

5b30322

hyeok9855 requested review from leonlan and fedebotu May 31, 2024 22:20

leonlan reviewed Jun 1, 2024

View reviewed changes

rl4co/envs/routing/cvrp/local_search.py Outdated Show resolved Hide resolved

fedebotu reviewed Jun 1, 2024

View reviewed changes

hyeok9855 added 4 commits June 3, 2024 13:27

refactor following PR reviews

e5bca03

fix PyVRP local search

e90fad0

add zero-padding after local search in cvrp, minor debugs

3d04ead

Merge branch 'main' into local-search

e289307

hyeok9855 merged commit 1e8ae13 into ai4co:main Jun 8, 2024
0 of 12 checks passed

fedebotu added this to the 0.5.0 milestone Jun 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feat] Add Local Search for Solution Improvement #140

[Feat] Add Local Search for Solution Improvement #140

hyeok9855 commented Mar 15, 2024 •

edited

Loading

fedebotu commented Mar 16, 2024

hyeok9855 commented May 1, 2024

hyeok9855 commented May 31, 2024

leonlan commented Jun 1, 2024

fedebotu Jun 1, 2024

hyeok9855 Jun 1, 2024 •

edited

Loading

fedebotu Jun 1, 2024

hyeok9855 Jun 8, 2024

fedebotu Jun 1, 2024

fedebotu Jun 1, 2024

fedebotu Jun 1, 2024

hyeok9855 Jun 1, 2024

fedebotu Jun 1, 2024

hyeok9855 Jun 3, 2024

hyeok9855 Jun 3, 2024

fedebotu Jun 3, 2024

fedebotu commented Jun 1, 2024

hyeok9855 commented Jun 8, 2024 •

edited

Loading


		return heatmaps_logits
		heatmap += 1e-10 if heatmap.dtype != torch.float16 else 3e-8

[Feat] Add Local Search for Solution Improvement #140

[Feat] Add Local Search for Solution Improvement #140

Conversation

hyeok9855 commented Mar 15, 2024 • edited Loading

Description

Motivation and Context

Types of changes

Checklist

fedebotu commented Mar 16, 2024

hyeok9855 commented May 1, 2024

Changes

hyeok9855 commented May 31, 2024

Changes

leonlan commented Jun 1, 2024

Choose a reason for hiding this comment

hyeok9855 Jun 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fedebotu commented Jun 1, 2024

hyeok9855 commented Jun 8, 2024 • edited Loading

hyeok9855 commented Mar 15, 2024 •

edited

Loading

hyeok9855 Jun 1, 2024 •

edited

Loading

hyeok9855 commented Jun 8, 2024 •

edited

Loading