Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Product sync: use OFF's Redis stream to get live updates #620

Open
raphodn opened this issue Dec 11, 2024 · 2 comments · Fixed by #618, #621, #622 or #707
Open

Product sync: use OFF's Redis stream to get live updates #620

raphodn opened this issue Dec 11, 2024 · 2 comments · Fixed by #618, #621, #622 or #707
Labels
Product data sync

Comments

@raphodn
Copy link
Member

raphodn commented Dec 11, 2024

Story

We currently sync all of OFF's product database every day (see #460 for more details).
But this system has some issues :

  • the sync can take over an 1h to run. every day
  • if a user makes changes to a product (or simply adds a new one), it will take sometimes 24h to get the result/feedback in OP

OFF has a stream where every product creation & modification is available. We could plug OP on this stream.
@raphodn raphodn changed the title Product sync: use OFF's Redis stream Product sync: use OFF's Redis stream to get live updates Dec 11, 2024
@raphodn raphodn linked a pull request Dec 11, 2024 that will close this issue
feat: listen to updates from Product Opener using Redis #618
Merged
@raphodn raphodn moved this from Backlog to In progress in 💸 Open Prices Dec 11, 2024
This was linked to pull requests Dec 12, 2024
chore: enable fetching product updates from Redis #621
Merged
fix: create a product if needed after a redis update event #622
Merged
@raphodn
Copy link
Member Author

raphodn commented Feb 2, 2025

Since #618 we now get live updates !

But only for OFF products I believe, not OBF nor OPF nor OPFF.

@raphodn
Copy link
Member Author

raphodn commented Feb 5, 2025 •
edited
Loading

I see some improvements :

  • disable OFF DB daily sync now that we have redis
  • we don't use the diffs that Redis sends us, but call the OFF API instead...
  • do we still need to call product_post_create_fetch_and_save_data_from_openfoodfacts each time a product is created in OP ? if it's already unknown in OP, then it won't be available in OFF (as we fetch only OFF data, not OxF)
  • can we get Redis live updates from OBF, OPF & OPFF ?

@raphodn raphodn linked a pull request Feb 5, 2025 that will close this issue
refactor(Product sync): disable OFF daily sync (now that we have Redis live updates) #707
Merged
@github-project-automation github-project-automation bot moved this from In progress to Done in 💸 Open Prices Feb 5, 2025
@raphodn raphodn reopened this Feb 5, 2025
@github-project-automation github-project-automation bot moved this from Done to Ready in 💸 Open Prices Feb 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Product data sync
Projects
💸 Open Prices
Status: Ready
Milestone
No milestone
1 participant
@raphodn