This code converts warc files into html and json files.
The Internet Archive is a non-profit digital library that aims to provide universal access to knowledge and preserve digital content for future generations
it is a format of compressed file containing web pages.
So if you don't have a file containing warc files you will need to download them first
which I will tell you how to download
However if you have warc files, you can skip this part and go directly to the requirements part
We will use commoncrawl website
commoncrawl
1)Firstly, we will download paths for the year required
Click ➡️:datasets
each path file is a text file conatining names of the compresssed warc files
2) Copy one line
3)Go to your browser then write https://data.commoncrawl.org/ then paste the line you copied and place it after the forward slash
4)Finally, after the download is complete you will have your compressed warc file
5)If you want more compressed warc files for the same year repeat steps 2 & 3 with more lines form the path text file