Index media, reply status, quote status and extended URL #393

SaptakS · 2025-02-06T12:04:45Z

This PR is still a work in progress. This PR saves media to the local filesytem, creates a table to store the media key, type and tweetID to link it with the correct tweets. Also updates the tweet table to have a hasMedia column.

Things left to do:

Store reply information
Store quote information
Figure out issue with downloading of videos
Save URL and expanded URL of URL entities
Implement above with archive import
Build all of this into the X archive that Cyd creates

- Saves mediaID, media type and tweetID in the media table - Saves hasMedia to true if there is media. Using hasMedia and tweetID in the media table, one can retrieve information about all the media associated with a tweet - Downloads and saves the actual media files

micahflee

This is great so far.

Can you write tests for it, and include blocks of JSON as testdata? So, in a test account, you can save an API response that includes tweets with media (and later, with replies, quote tweets, links, etc.) as JSON in the testdata folder, and then write tests that load it. (Also if you can make any improvements on the testing situation, that would be great...)

I think we should capture more info in the tweet_media table too. In my test account, I posted two tweets, one with an image and one without media. Here is the tweet with an image:

Even though the tweet does not have a link, here is the text field that is pulled from the JSON: a fun game https://t.co/FuXqivJIdE

Here is a piece of the entities[].media object:

{
    "display_url": "pic.x.com/FuXqivJIdE",
    "expanded_url": "https://x.com/snowyfoxmatch/status/1887628076441608562/photo/1",
    "id_str": "1887628033273831427",
    "indices": [
        11,
        34
    ],
    "media_key": "3_1887628033273831427",
    "media_url_https": "https://pbs.twimg.com/media/GjIzsfdbIAMxByO.jpg",
    "type": "photo",
    "url": "https://t.co/FuXqivJIdE",
    "ext_media_availability": {
        "status": "Available"
    },

Ultimately, when we repost this tweet to Bluesky or Mastodon, we want will want the text to just say: a fun game and to strip the https://t.co/FuXqivJIdE link. In order to do this, we will need to save url as well. (And potentially the values in indices, as these will tell us where in the text string the link is, though we can also find that by searching the string.)

Here's the full JSON for my test tweets btw, which you could use for testing if you want:

{"data":{"user":{"result":{"__typename":"User","timeline_v2":{"timeline":{"instructions":[{"type":"TimelineClearCache"},{"type":"TimelineAddEntries","entries":[{"entryId":"tweet-1887628109589164504","sortIndex":"1887628966789906432","content":{"entryType":"TimelineTimelineItem","__typename":"TimelineTimelineItem","itemContent":{"itemType":"TimelineTweet","__typename":"TimelineTweet","tweet_results":{"result":{"__typename":"Tweet","rest_id":"1887628109589164504","core":{"user_results":{"result":{"__typename":"User","id":"VXNlcjoxODMzNjUxMDUzMTY4MDcwNjU2","rest_id":"1833651053168070656","affiliates_highlighted_label":{},"has_graduated_access":false,"parody_commentary_fan_label":"None","is_blue_verified":false,"profile_image_shape":"Circle","legacy":{"protected":true,"following":false,"can_dm":true,"can_media_tag":false,"created_at":"Tue Sep 10 23:37:56 +0000 2024","default_profile":true,"default_profile_image":false,"description":"","entities":{"description":{"urls":[]}},"fast_followers_count":0,"favourites_count":0,"followers_count":0,"friends_count":0,"has_custom_timelines":false,"is_translator":false,"listed_count":0,"location":"","media_count":1,"name":"snowyfoxmatch","needs_phone_verification":false,"normal_followers_count":0,"pinned_tweet_ids_str":[],"possibly_sensitive":false,"profile_image_url_https":"https://pbs.twimg.com/profile_images/1833667031700586496/a_XCrxOt_normal.jpg","profile_interstitial_type":"","screen_name":"snowyfoxmatch","statuses_count":2,"translator_type":"none","verified":false,"want_retweets":false,"withheld_in_countries":[]},"tipjar_settings":{}}}},"unmention_data":{},"edit_control":{"edit_tweet_ids":["1887628109589164504"],"editable_until_msecs":"1738884186000","is_edit_eligible":true,"edits_remaining":"5"},"is_translatable":false,"views":{"state":"Enabled"},"source":"<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>","grok_analysis_button":true,"legacy":{"bookmark_count":0,"bookmarked":false,"created_at":"Thu Feb 06 22:23:06 +0000 2025","conversation_id_str":"1887628109589164504","display_text_range":[0,20],"entities":{"hashtags":[],"symbols":[],"timestamps":[],"urls":[],"user_mentions":[]},"favorite_count":0,"favorited":false,"full_text":"a post without media","is_quote_status":false,"lang":"en","quote_count":0,"reply_count":0,"retweet_count":0,"retweeted":false,"user_id_str":"1833651053168070656","id_str":"1887628109589164504"},"quick_promote_eligibility":{"eligibility":"IneligibleNotProfessional"}}},"tweetDisplayType":"Tweet"},"clientEventInfo":{"component":"tweet","element":"tweet","details":{"timelinesDetails":{"injectionType":"RankedOrganicTweet","controllerData":"DAACDAABDAABCgABAAAAAAAAAAAKAAkZcm/q8hqQAAAAAAA="}}}}},{"entryId":"tweet-1887628076441608562","sortIndex":"1887628966789906431","content":{"entryType":"TimelineTimelineItem","__typename":"TimelineTimelineItem","itemContent":{"itemType":"TimelineTweet","__typename":"TimelineTweet","tweet_results":{"result":{"__typename":"Tweet","rest_id":"1887628076441608562","core":{"user_results":{"result":{"__typename":"User","id":"VXNlcjoxODMzNjUxMDUzMTY4MDcwNjU2","rest_id":"1833651053168070656","affiliates_highlighted_label":{},"has_graduated_access":false,"parody_commentary_fan_label":"None","is_blue_verified":false,"profile_image_shape":"Circle","legacy":{"protected":true,"following":false,"can_dm":true,"can_media_tag":false,"created_at":"Tue Sep 10 23:37:56 +0000 2024","default_profile":true,"default_profile_image":false,"description":"","entities":{"description":{"urls":[]}},"fast_followers_count":0,"favourites_count":0,"followers_count":0,"friends_count":0,"has_custom_timelines":false,"is_translator":false,"listed_count":0,"location":"","media_count":1,"name":"snowyfoxmatch","needs_phone_verification":false,"normal_followers_count":0,"pinned_tweet_ids_str":[],"possibly_sensitive":false,"profile_image_url_https":"https://pbs.twimg.com/profile_images/1833667031700586496/a_XCrxOt_normal.jpg","profile_interstitial_type":"","screen_name":"snowyfoxmatch","statuses_count":2,"translator_type":"none","verified":false,"want_retweets":false,"withheld_in_countries":[]},"tipjar_settings":{}}}},"unmention_data":{},"edit_control":{"edit_tweet_ids":["1887628076441608562"],"editable_until_msecs":"1738884178000","is_edit_eligible":true,"edits_remaining":"5"},"is_translatable":false,"views":{"state":"Enabled"},"source":"<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>","grok_analysis_button":true,"legacy":{"bookmark_count":0,"bookmarked":false,"created_at":"Thu Feb 06 22:22:58 +0000 2025","conversation_id_str":"1887628076441608562","display_text_range":[0,10],"entities":{"hashtags":[],"media":[{"display_url":"pic.x.com/FuXqivJIdE","expanded_url":"https://x.com/snowyfoxmatch/status/1887628076441608562/photo/1","id_str":"1887628033273831427","indices":[11,34],"media_key":"3_1887628033273831427","media_url_https":"https://pbs.twimg.com/media/GjIzsfdbIAMxByO.jpg","type":"photo","url":"https://t.co/FuXqivJIdE","ext_media_availability":{"status":"Available"},"features":{"large":{"faces":[{"x":492,"y":390,"h":34,"w":34},{"x":355,"y":141,"h":53,"w":53}]},"medium":{"faces":[{"x":492,"y":390,"h":34,"w":34},{"x":355,"y":141,"h":53,"w":53}]},"small":{"faces":[{"x":446,"y":353,"h":30,"w":30},{"x":321,"y":127,"h":48,"w":48}]},"orig":{"faces":[{"x":492,"y":390,"h":34,"w":34},{"x":355,"y":141,"h":53,"w":53}]}},"sizes":{"large":{"h":502,"w":750,"resize":"fit"},"medium":{"h":502,"w":750,"resize":"fit"},"small":{"h":455,"w":680,"resize":"fit"},"thumb":{"h":150,"w":150,"resize":"crop"}},"original_info":{"height":502,"width":750,"focus_rects":[{"x":0,"y":82,"w":750,"h":420},{"x":248,"y":0,"w":502,"h":502},{"x":310,"y":0,"w":440,"h":502},{"x":499,"y":0,"w":251,"h":502},{"x":0,"y":0,"w":750,"h":502}]},"allow_download_status":{"allow_download":true},"media_results":{"result":{"media_key":"3_1887628033273831427"}}}],"symbols":[],"timestamps":[],"urls":[],"user_mentions":[]},"extended_entities":{"media":[{"display_url":"pic.x.com/FuXqivJIdE","expanded_url":"https://x.com/snowyfoxmatch/status/1887628076441608562/photo/1","id_str":"1887628033273831427","indices":[11,34],"media_key":"3_1887628033273831427","media_url_https":"https://pbs.twimg.com/media/GjIzsfdbIAMxByO.jpg","type":"photo","url":"https://t.co/FuXqivJIdE","ext_media_availability":{"status":"Available"},"features":{"large":{"faces":[{"x":492,"y":390,"h":34,"w":34},{"x":355,"y":141,"h":53,"w":53}]},"medium":{"faces":[{"x":492,"y":390,"h":34,"w":34},{"x":355,"y":141,"h":53,"w":53}]},"small":{"faces":[{"x":446,"y":353,"h":30,"w":30},{"x":321,"y":127,"h":48,"w":48}]},"orig":{"faces":[{"x":492,"y":390,"h":34,"w":34},{"x":355,"y":141,"h":53,"w":53}]}},"sizes":{"large":{"h":502,"w":750,"resize":"fit"},"medium":{"h":502,"w":750,"resize":"fit"},"small":{"h":455,"w":680,"resize":"fit"},"thumb":{"h":150,"w":150,"resize":"crop"}},"original_info":{"height":502,"width":750,"focus_rects":[{"x":0,"y":82,"w":750,"h":420},{"x":248,"y":0,"w":502,"h":502},{"x":310,"y":0,"w":440,"h":502},{"x":499,"y":0,"w":251,"h":502},{"x":0,"y":0,"w":750,"h":502}]},"allow_download_status":{"allow_download":true},"media_results":{"result":{"media_key":"3_1887628033273831427"}}}]},"favorite_count":0,"favorited":false,"full_text":"a fun game https://t.co/FuXqivJIdE","is_quote_status":false,"lang":"en","possibly_sensitive":false,"possibly_sensitive_editable":true,"quote_count":0,"reply_count":0,"retweet_count":0,"retweeted":false,"user_id_str":"1833651053168070656","id_str":"1887628076441608562"},"quick_promote_eligibility":{"eligibility":"IneligibleNotProfessional"}}},"tweetDisplayType":"Tweet"},"clientEventInfo":{"component":"tweet","element":"tweet","details":{"timelinesDetails":{"injectionType":"RankedOrganicTweet","controllerData":"DAACDAABDAABCgABAAAAAAAAAAAKAAkZcm/q8hqQAAAAAAA="}}}}},{"entryId":"who-to-follow-1887628966789906434","sortIndex":"1887628966789906430","content":{"entryType":"TimelineTimelineModule","__typename":"TimelineTimelineModule","items":[{"entryId":"who-to-follow-1887628966789906434-user-10671602","item":{"itemContent":{"itemType":"TimelineUser","__typename":"TimelineUser","user_results":{"result":{"__typename":"User","id":"VXNlcjoxMDY3MTYwMg==","rest_id":"10671602","affiliates_highlighted_label":{},"has_graduated_access":true,"parody_commentary_fan_label":"None","is_blue_verified":true,"profile_image_shape":"Square","legacy":{"following":false,"can_dm":false,"can_media_tag":false,"created_at":"Tue Nov 27 22:37:20 +0000 2007","default_profile":false,"default_profile_image":false,"description":"Official Sony Interactive Entertainment account. Updates on PS5, PlayStation VR2, PlayStation Plus, PS4 and more. Support: @AskPlayStation","entities":{"description":{"urls":[]},"url":{"urls":[{"display_url":"playstation.com","expanded_url":"https://www.playstation.com","url":"https://t.co/OrydDMkek8","indices":[0,23]}]}},"fast_followers_count":0,"favourites_count":1453,"followers_count":42345013,"friends_count":752,"has_custom_timelines":true,"is_translator":false,"listed_count":34426,"location":"California","media_count":26512,"name":"PlayStation","normal_followers_count":42345013,"pinned_tweet_ids_str":["1876618946146677236"],"possibly_sensitive":false,"profile_banner_url":"https://pbs.twimg.com/profile_banners/10671602/1738705748","profile_image_url_https":"https://pbs.twimg.com/profile_images/1833447364138299392/AXIZsQe4_normal.jpg","profile_interstitial_type":"","screen_name":"PlayStation","statuses_count":46642,"translator_type":"none","url":"https://t.co/OrydDMkek8","verified":false,"verified_type":"Business","want_retweets":false,"withheld_in_countries":[]},"tipjar_settings":{}}},"userDisplayType":"User"},"clientEventInfo":{"component":"suggest_who_to_follow","element":"user","details":{"timelinesDetails":{"injectionType":"WhoToFollow","sourceData":"DAABCgABHlvecZFOhsEKAAIAAAAAAAAAAAAIAAIAAARgCAADAAAAAgA="}}}}},{"entryId":"who-to-follow-1887628966789906434-user-24742040","item":{"itemContent":{"itemType":"TimelineUser","__typename":"TimelineUser","user_results":{"result":{"__typename":"User","id":"VXNlcjoyNDc0MjA0MA==","rest_id":"24742040","affiliates_highlighted_label":{},"has_graduated_access":true,"parody_commentary_fan_label":"None","is_blue_verified":true,"profile_image_shape":"Square","legacy":{"following":false,"can_dm":false,"can_media_tag":false,"created_at":"Mon Mar 16 18:30:52 +0000 2009","default_profile":false,"default_profile_image":false,"description":"Are you telling me an X made this Box?","entities":{"description":{"urls":[]},"url":{"urls":[{"display_url":"Xbox.com","expanded_url":"http://Xbox.com","url":"https://t.co/C1bIH5PDZK","indices":[0,23]}]}},"fast_followers_count":0,"favourites_count":6956,"followers_count":24680499,"friends_count":15052,"has_custom_timelines":true,"is_translator":false,"listed_count":21721,"location":"Redmond, Washington","media_count":44234,"name":"Xbox","normal_followers_count":24680499,"pinned_tweet_ids_str":["1887260502482616761"],"possibly_sensitive":false,"profile_banner_url":"https://pbs.twimg.com/profile_banners/24742040/1738864839","profile_image_url_https":"https://pbs.twimg.com/profile_images/1885376956990185478/5pKp_1Ti_normal.jpg","profile_interstitial_type":"","screen_name":"Xbox","statuses_count":316697,"translator_type":"none","url":"https://t.co/C1bIH5PDZK","verified":false,"verified_type":"Business","want_retweets":false,"withheld_in_countries":[]},"tipjar_settings":{}}},"userDisplayType":"User"},"clientEventInfo":{"component":"suggest_who_to_follow","element":"user","details":{"timelinesDetails":{"injectionType":"WhoToFollow","sourceData":"DAABCgABHlvecZFOhsEKAAIAAAAAAAAAAAAIAAIAAARgCAADAAAAAgA="}}}}},{"entryId":"who-to-follow-1887628966789906434-user-11928542","item":{"itemContent":{"itemType":"TimelineUser","__typename":"TimelineUser","user_results":{"result":{"__typename":"User","id":"VXNlcjoxMTkyODU0Mg==","rest_id":"11928542","affiliates_highlighted_label":{},"has_graduated_access":true,"parody_commentary_fan_label":"None","is_blue_verified":true,"profile_image_shape":"Circle","legacy":{"following":false,"can_dm":false,"can_media_tag":false,"created_at":"Mon Jan 07 04:09:59 +0000 2008","default_profile":false,"default_profile_image":false,"description":"Play Dark Cloud 2","entities":{"description":{"urls":[]},"url":{"urls":[{"display_url":"kotaku.com","expanded_url":"http://kotaku.com","url":"https://t.co/gMN9Ce6bJ3","indices":[0,23]}]}},"fast_followers_count":0,"favourites_count":227,"followers_count":2632714,"friends_count":199,"has_custom_timelines":true,"is_translator":false,"listed_count":12590,"location":"","media_count":83900,"name":"Kotaku","normal_followers_count":2632714,"pinned_tweet_ids_str":[],"possibly_sensitive":false,"profile_banner_url":"https://pbs.twimg.com/profile_banners/11928542/1559318957","profile_image_url_https":"https://pbs.twimg.com/profile_images/1145899315006717952/ozxwJgmx_normal.png","profile_interstitial_type":"","screen_name":"Kotaku","statuses_count":119397,"translator_type":"none","url":"https://t.co/gMN9Ce6bJ3","verified":false,"want_retweets":false,"withheld_in_countries":[]},"tipjar_settings":{}}},"userDisplayType":"User"},"clientEventInfo":{"component":"suggest_who_to_follow","element":"user","details":{"timelinesDetails":{"injectionType":"WhoToFollow","sourceData":"DAABCgABHlvecZFOhsEKAAIAAAAAAAAAAAAIAAIAAARgCAADAAAAAgA="}}}}}],"displayType":"Vertical","header":{"displayType":"Classic","text":"Who to follow","sticky":false},"footer":{"displayType":"Classic","text":"Show more","landingUrl":{"url":"twitter://connect_people?user_id=1833651053168070656&display_location=profile_wtf_showmore","urlType":"DeepLink"}},"clientEventInfo":{"component":"suggest_who_to_follow","details":{"timelinesDetails":{"injectionType":"WhoToFollow","sourceData":"DAABCgABHlvecZFOhsEKAAIAAAAAAAAAAAAIAAIAAARgCAADAAAAAgA="}}}}},{"entryId":"cursor-top-1887628966789906433","sortIndex":"1887628966789906433","content":{"entryType":"TimelineTimelineCursor","__typename":"TimelineTimelineCursor","value":"DAABCgABGjI0i1FAJxEKAAIaMjPDvBqx2AgAAwAAAAEAAA","cursorType":"Top"}},{"entryId":"cursor-bottom-1887628966789906429","sortIndex":"1887628966789906429","content":{"entryType":"TimelineTimelineCursor","__typename":"TimelineTimelineCursor","value":"DAABCgABGjI0i1E___sKAAIaMjO8BFshcggAAwAAAAIAAA","cursorType":"Bottom"}}]}],"metadata":{"scribeConfig":{"page":"profileBest"}}}}}}}}

src/account_x/x_account_controller.ts

SaptakS · 2025-02-07T06:35:11Z

Yes, the media entities does have a lot of information, do you think we should probably capture all the informations like expanded_url, etc.? Or just url and indices should be enough?

…ions

micahflee · 2025-02-07T17:15:14Z

Yes, the media entities does have a lot of information, do you think we should probably capture all the informations like expanded_url, etc.? Or just url and indices should be enough?

I think url and indices should be enough. Looks like expanded_url is the link for if you click on the image and it shows it to you big in the web app, so I don't think we care about that.

…to use camelCase for indexStart and indexEnd for consistency

…rectly

…Ls from the db before indexing if they have previously been indexed

SaptakS · 2025-02-17T15:34:50Z

@micahflee added the importing information from archive part . Quote informations are not there in archives, so left those out.

Support for indexing video (and fix indexing URLs)

…ed it ourselves

…es media list, not the length of it. Rename start_index and end_index to startIndex and endIndex. And when importing URLs, delete the URL first if it already exists to prevent unique contraint error.

…r links

Media in Cyd's local X archive

…t already exists to make sure we get media/links/etc

…ent PR

… it, and then the import fails because it is not looking in the folder it unzipped into

micahflee · 2025-02-17T22:12:56Z

I just merged #402 into this branch, so now we only have one PR for this feature for you to review @redshiftzero.

I fixed an existing bug where when you import an X archive from a folder (not a zip file), it would delete the original X archive. Now it should only delete the temp folder if it unzips it for you.

When testing, make sure you have an X account that includes tweets with images, videos, URLs, and quote tweets. Some things to test:

Build an archive from scratch
- Then check the database and bake sure the tweet table has reply and quoted tweet fields.
- Make sure the tweet_media table has the right media, and that the file with name filename is in your Cyd archive "Tweet Media" folder.
- Make sure the tweet_url has links.
- Open the Cyd archive and make sure all the media and links and quote tweets display as you'd expect them to.
Delete your ~/Documents/Cyd Dev/X/$USERNAME folder, and then import an X archive. Check all the same things as above -- though, the archive doesn't include quote tweet data, so those will just be links.
Delete your ~/Application Support/Cyd Dev and ~/Documents/Cyd Dev/X/$USERNAME folders, switch to the main branch, and then import your X archive and view it. It should be the old version -- only t.co links, no media, etc. Then switch back to this branch and import your X archive again. It should update all of your old info instead of skipping it.

SaptakS · 2025-02-18T14:00:08Z

@micahflee thanks for taking care of the bugs! Looks good to me.

redshiftzero · 2025-02-19T02:55:32Z

Tested these three scenarios:

1. Build archive from scratch ✅

All looks good! Replies, video, quote tweet all good.

2. Build archive from X export ❌

One issue: videos didn't appear either in the "Tweet Media" folder or in the Cyd archive UI. This did work for scenario 1.

(Note: I didn't test replies in step 2 as my X archive didn't have them as I only added replies for review of this PR an few hours ago)

3. Upgrade ✅

Tested with archive created on main (deleting ~/Library/Application Support/Cyd Dev and ~/Documents/Cyd Dev/X/$USERNAME) and then updating using this branch

All looks good!

redshiftzero · 2025-02-19T03:28:35Z

src/account_x/x_account_controller.ts

+
+        // If file doesn't exist in archive, don't save information in db
+        if (!fs.existsSync(archiveMediaFilename)) {
+            log.info(`XAccountController.saveXArchiveMedia: media file not found: ${archiveMediaFilename}`);


with the videos for loading from the export, I'm hitting this log line here

this is the log line:

22:27:32.890 › XAccountController.saveXArchiveMedia: media file not found: /Users/redshiftzero/Documents/Cyd Dev/X/$TEST_ACCOUNT_USERNAME/tmp/data/tweets_media/1891594792125010175-kvvBMojc1QwudY9d.mp4

the file in question is named data/tweets_media/1891594792125010175-jq7vI73puwizA_En.mp4 in the archive

In the media object, there are various bitrate options, and it seems like the one by default in the archive is the highest bitrate one

added 7deb52c which fixes for me

redshiftzero

added a small commit resolving the video import from the X archive file, all LGTM now!

SaptakS added 3 commits February 6, 2025 16:39

Adds types and migrations for tweet_media table

b5e762e

Adds reply and quote related informations to the tweet table

8708243

micahflee reviewed Feb 6, 2025

View reviewed changes

src/account_x/x_account_controller.ts Outdated Show resolved Hide resolved

src/account_x/x_account_controller.ts Outdated Show resolved Hide resolved

Adds additional fields to tweet_media to store url and index informat…

41fc9a0

…ions

SaptakS force-pushed the 354-index-media branch from e21caf3 to 1b36b14 Compare February 7, 2025 15:09

Index tweet URLs in the database

b2bf23f

SaptakS force-pushed the 354-index-media branch from 1b36b14 to b2bf23f Compare February 12, 2025 13:56

micahflee added 3 commits February 14, 2025 14:14

Add new types for tweet media, and support downloading videos

870f333

Add more fields to the XTweetMediaRow type, and update the migration …

e4c7a5c

…to use camelCase for indexStart and indexEnd for consistency

Add a test that ensures tweets with videos and photos are indexed cor…

5f49465

…rectly

micahflee mentioned this pull request Feb 14, 2025

Support for indexing video (and fix indexing URLs) #401

Merged

micahflee and others added 5 commits February 16, 2025 10:33

Fix indexing URLs, and write test for it

ca55615

Add media and URLs to archive.js in buildArchive; and also, delete UR…

217ed5e

…Ls from the db before indexing if they have previously been indexed

Update x-archive deps

76e849f

Properly display links and media in X archive

dd6564a

Imports media, urls, and reply information from tweet archives

44e89ac

SaptakS added 2 commits February 17, 2025 23:52

Merge pull request #401 from lockdown-systems/354-index-media-video

d53a26b

Support for indexing video (and fix indexing URLs)

Adds types to the archive import functions

21dba91

SaptakS force-pushed the 354-index-media branch from 2998517 to 21dba91 Compare February 17, 2025 18:31

micahflee and others added 5 commits February 17, 2025 11:14

Make quote tweets work in the archive

dbfbdad

Simplify quote tweets

acaba64

Merge branch '354-index-media' into 354-index-media-archive

df22fed

Merge branch 'main' into 354-index-media

7107144

Merge branch '354-index-media' into 354-index-media-archive

c2b387b

micahflee marked this pull request as ready for review February 17, 2025 19:28

Fix INSERT query when importing tweets -- there were 20 ?s but 15 params

045dfa9

micahflee and others added 8 commits February 17, 2025 11:39

When importing X archive, only delete the archive folder if we unzipp…

d52675c

…ed it ourselves

Fix importing media items so it sets mediaList to the extended entiti…

fab76b9

…es media list, not the length of it. Rename start_index and end_index to startIndex and endIndex. And when importing URLs, delete the URL first if it already exists to prevent unique contraint error.

Switch to using extended_entities for media, and just use entities fo…

f2863db

…r links

Merge pull request #402 from lockdown-systems/354-index-media-archive

19a1e45

Media in Cyd's local X archive

When importing tweets from an X archive, always delete the tweet if i…

72f78f0

…t already exists to make sure we get media/links/etc

Remove dead code

c957404

Remove IPC functions from X that should have been removed in a differ…

2e71e52

…ent PR

Fix bug where when you import an X archive from a ZIP file, it unzips…

47f9fde

… it, and then the import fails because it is not looking in the folder it unzipped into

micahflee mentioned this pull request Feb 18, 2025

Implement migrate to Bluesky #371

Merged

12 tasks

redshiftzero self-requested a review February 19, 2025 00:56

redshiftzero reviewed Feb 19, 2025

View reviewed changes

fix: load videos from X archive export

7deb52c

redshiftzero approved these changes Feb 19, 2025

View reviewed changes

micahflee merged commit bab4c53 into main Feb 19, 2025
1 check passed

micahflee deleted the 354-index-media branch February 19, 2025 03:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Index media, reply status, quote status and extended URL #393

Index media, reply status, quote status and extended URL #393

SaptakS commented Feb 6, 2025 •

edited by micahflee

Loading

micahflee left a comment

SaptakS commented Feb 7, 2025

micahflee commented Feb 7, 2025

SaptakS commented Feb 17, 2025

micahflee commented Feb 17, 2025

SaptakS commented Feb 18, 2025

redshiftzero commented Feb 19, 2025 •

edited

Loading

redshiftzero Feb 19, 2025

redshiftzero Feb 19, 2025

redshiftzero Feb 19, 2025

redshiftzero Feb 19, 2025

redshiftzero left a comment

Index media, reply status, quote status and extended URL #393

Index media, reply status, quote status and extended URL #393

Conversation

SaptakS commented Feb 6, 2025 • edited by micahflee Loading

micahflee left a comment

Choose a reason for hiding this comment

SaptakS commented Feb 7, 2025

micahflee commented Feb 7, 2025

SaptakS commented Feb 17, 2025

micahflee commented Feb 17, 2025

SaptakS commented Feb 18, 2025

redshiftzero commented Feb 19, 2025 • edited Loading

1. Build archive from scratch ✅

2. Build archive from X export ❌

3. Upgrade ✅

redshiftzero Feb 19, 2025

Choose a reason for hiding this comment

redshiftzero Feb 19, 2025

Choose a reason for hiding this comment

redshiftzero Feb 19, 2025

Choose a reason for hiding this comment

redshiftzero Feb 19, 2025

Choose a reason for hiding this comment

redshiftzero left a comment

Choose a reason for hiding this comment

SaptakS commented Feb 6, 2025 •

edited by micahflee

Loading

redshiftzero commented Feb 19, 2025 •

edited

Loading