Scraping of market_value_history is broken

[dcaribou]

Parent issue → dcaribou/transfermarkt-datasets#215

The attribute market_value_history is coming as null in the latest runs, likely due to an upstream change on Transfermarkt side.

$ scrapy crawl players -a parents=samples/clubs.json -s USER_AGENT="..." | jq '.market_value_history'
null
null
null
...

[dcaribou]

The html used for extracting the market_value_history has changed significantly on a recent update in Transfermarkt, and the existing logic for extracting the data does no longer work

transfermarkt-scraper/tfmkt/spiders/players.py

Line 120 in 3e4ccb8

def parse_market_history(self, response: Response):

The new html uses an svg graph which appears quite hard to reverse-engineer at this
point.

[LarchLiu]

I found that there will be a request on this page, so maybe that helps a bit.

Request URL:
https://www.transfermarkt.com/ceapi/marketValueDevelopment/graph/28003

[dcaribou]

Ah, and it's an open API 🙌
This is super helpful, even the response format is the same as the scraped market_value_history object.

[LarchLiu]

Should we create a new player_valuations crawler with this API?
Seems like it just in development?

[n-richaud]

the old graph is still available on this page : https://www.transfermarkt.fr/dimitri-payet/marktwertverlauf/spieler/37647