Fix anonymous fetch of users and posts #2533

SamuMazzi · 2025-03-13T22:54:05Z

Hello! This PR aims to fix the problem that caused many and many users to be unable to download anything, either profiles and posts.

This problem affected a lot of us, but in the end, there was simply some backend change made by Instagram, so it was needed to change some parameters and adapt to the new responses, and now some things like _convert_iphone_carousel are not even needed anymore.

Actually there are way more incompatible changes (i.e. logged in user doesn't work, neither the fetching of users by their id) so someone should take care also of that.

Is it just a proof of concept? No
Is the documentation updated (if appropriate)? No
Do you consider it ready to be merged or is it a draft? Ready
Can we help you at some point? No

I think I made all the changes correctly, but a deeper look from a more experienced "instaloader developer" would be appreciated, thanks!

Fixes #2532, #2528, #2307, #2517, #2501, #2511, #2508

anaacaren · 2025-03-15T10:57:14Z

Hi! I've just ran your changes, it works like a charm!! Thank you :)

kakaroto · 2025-03-15T22:51:20Z

I tried your branch as well, and it works great. Though it seems that I still get the 401 unauthorized errors when it loads the session from file, so it asks me for the password every time.

Loaded session from C:\Users\kakaroto\AppData\Local\Instaloader\session-XXXXX.
JSON Query to graphql/query: 401 Unauthorized - "fail" status, message "Please wait a few minutes before you try again." when accessing https://www.instagram.com/graphql/query?query_hash=d6f4427fbe92d846298cf93df0b937d3&variables=%7B%7D [retrying; skip with ^C]
JSON Query to graphql/query: 401 Unauthorized - "fail" status, message "Please wait a few minutes before you try again." when accessing https://www.instagram.com/graphql/query?query_hash=d6f4427fbe92d846298cf93df0b937d3&variables=%7B%7D [retrying; skip with ^C]
Error when checking if logged in: JSON Query to graphql/query: 401 Unauthorized - "fail" status, message "Please wait a few minutes before you try again." when accessing https://www.instagram.com/graphql/query?query_hash=d6f4427fbe92d846298cf93df0b937d3&variables=%7B%7D
Enter Instagram password for XXXXX:

Not sure if it's related to the same core issue you sent the PR for, or if it's an unrelated issue.

SamuMazzi · 2025-03-16T23:47:25Z

@kakaroto yeah I know, this PR doesn't aim to solve that problem yet... I'm not really using that functionality so I don't know if/when I'll fix it, sorry!

kakaroto · 2025-03-17T02:59:33Z

@kakaroto yeah I know, this PR doesn't aim to solve that problem yet... I'm not really using that functionality so I don't know if/when I'll fix it, sorry!

No worries! I'll try to give it a poke at some point if I get the time. Thanks for your changes, it helps already!

jaybiirds · 2025-03-20T01:41:02Z

I forked the repository and pushed your commit and it seemed to work great, but after 96 posts downloaded, I got the same JSON Query to graphql/query: 401 Unauthorized - "fail" status, message "Please wait a few minutes before you try again." error. None of my JSON files are readable either, unsure if I did something wrong or if picking an account with a lot of posts had something to do with it.

xathian · 2025-03-22T08:27:52Z

I forked the repository and pushed your commit and it seemed to work great, but after 96 posts downloaded, I got the same JSON Query to graphql/query: 401 Unauthorized - "fail" status, message "Please wait a few minutes before you try again." error. None of my JSON files are readable either, unsure if I did something wrong or if picking an account with a lot of posts had something to do with it.

Unfortunately I can't say for certain but I am pretty sure I downloaded an account with over 96 posts last week just fine using these fixes.

xathian · 2025-03-23T21:38:17Z

This PR was working perfect for me yesterday but now I'm getting this error again:

JSON Query to api/v1/users/web_profile_info/?username=username_username: 401 Unauthorized - "fail" status, message "Please wait a few minutes before you try again." when accessing https://i.instagram.com/api/v1/users/web_profile_info/?username=username_username [retrying; skip with ^C]

Can't download any posts at all. After I run it, it buffers for a few seconds, sends two copies of that error, then throws this error:

Error accessing Instagram profile: JSON Query to api/v1/users/web_profile_info/?username=username_username: 401 Unauthorized - "fail" status, message "Please wait a few minutes before you try again." when accessing https://i.instagram.com/api/v1/users/web_profile_info/?username=username_username

Anyone else having this issue?

Nope, just ran an update on an account and it grabbed 4 missing posts just fine

Plaoo · 2025-03-24T13:58:48Z

Works, thank you

latinokodi · 2025-03-28T14:31:20Z

Newbie question here, I've been trying to use this PR but I don't think I really know how to, could you be so kind and provide the commands needed to download this corrected version?

SamuMazzi · 2025-03-28T22:03:43Z

Hi @latinokodi, I don't know if there are better ways to do this but what I would do is cloning the repo from which I've created the modifications and then install it in your project

git clone https://github.com/SamuMazzi/instaloader.git
cd path/to/your_project
pip install path/to/instaloader

Hope it can be helpful!

willianbriotto · 2025-04-07T16:47:28Z

Works, thank you

latinokodi · 2025-04-08T12:34:03Z

Works, thank you

How did you make it work?

xcaptain · 2025-04-29T09:31:44Z

no longer works

gabriele2605 · 2025-05-13T18:03:24Z

Now needs at least the login even for open profiles

Exception has occurred: LoginRequiredException
Redirected to login page. Use --login.

IgorArnaut · 2025-05-22T15:51:21Z

It stopped working. Damn Meta.

VolpiGiuliano · 2025-06-29T17:12:30Z

Hi @latinokodi, I don't know if there are better ways to do this but what I would do is cloning the repo from which I've created the modifications and then install it in your project
git clone https://github.com/SamuMazzi/instaloader.git
cd path/to/your_project
pip install path/to/instaloader
Hope it can be helpful!

It works!!!

aandergr · 2025-07-26T06:28:25Z

Thanks for the contribution — we’ll be happy to review it. Two observations upfront:

This PR changes the user agent, which in the past has affected the structure of returned data. Such a change is not without risk of breaking things. Therefore, this should be based on the upcoming/v4.15 branch, where related changes have already started.
from_iphone_struct() appears to no longer handle the structure we currently treat as iPhone-format. If there’s a new format, it should be handled in a separate constructor to avoid breaking existing functionality depending on from_iphone_struct().

Looking forward to the updated version.

Speedy1991 · 2025-08-14T15:01:35Z

Hey @aandergr,
@SamuMazzi has already done the heavy lifting on this one 🎉.
Would you have a chance to help bring it over the finish line? There are already several installed forks with this fix, but since it’s quite an important bug, it would be great to have it addressed upstream.

If there’s already an upcoming/v4.15 branch in the works, maybe we could merge these changes into it so it’s part of the next release?

SamuMazzi · 2025-08-28T18:00:58Z

@aandergr sorry for replying so late! Thank you for your response.. I think I did what you said but I'm not 100% so feel free to tell me what you think
(Also I took the freedom to update gitignore and change a small thing in the last commit, hope it's not a problem)

ekalin · 2025-08-28T18:03:18Z

The "new" iPhone struct actually looks like the old (non-iPhone) struct, created with the constructor.

billbeans · 2025-09-29T08:15:11Z

Hey, I tried your PR and I actually was able to download a profile for the first time in ages. The only thing I'm confused about is my terminal is littered with these error messages:

[ 2/48] **[REDACTED]**/2025-09-03_**-**-**_UTC_1.jpg download_pic(): Invalid URL '': No scheme supplied. Perhaps you meant https://? [retrying; skip with ^C]
download_pic(): Invalid URL '': No scheme supplied. Perhaps you meant https://? [retrying; skip with ^C]
Download <Post **[REDACTED]**> of **[REDACTED]**: download_pic(): Invalid URL '': No scheme supplied. Perhaps you meant https://?

Update: I tried another profile and now I'm suddenly not getting those errors. Weird

billbeans · 2025-10-30T03:17:46Z

Upon further usage, I've come to suspect that those errors are related to a change on Instagram's backend sometime around 2020. Older posts don't seem to yield those errors.

Seeing that the main branch still doesn't work for anonymous fetching, would you also be able to take a look at fetching ~~reels~~ highlights? They are completely unable to be scraped.

aandergr · 2025-11-08T16:08:33Z

Hi. I had finally the chance to have a deeper look into this. Thanks for your patience and for sharing your work on this!

The staticmethod Post::from_new_iphone_struct is actually not needed. The structure that is returned is not a "new iphone structure", it is rather the classical structure that the Post constructor parses. So replacing the Post.from_new_iphone_struct() invocations with Post() seems the way to go.

It attempts to download sidecar node videos for sidecar nodes that are no videos. This is easily fixed by rewriting the if condition more careful in instaloader.py:748: if video_url and sidecar_node.is_video and self.download_videos. (update: there is a change that explicitly sets it to an empty string instead of None, this needs to be reverted)

The PR lists some unrelated commits and unrelated changes. Unfortunately we cannot merge it as is. It would help us a lot if you could rebase it properly to upcoming/v4.15 and also remove the unrelated changes. (update: we could change the base in the PR)

With these changes, downloading anonymously seems fixed for profiles.

aandergr · 2025-11-08T17:26:17Z

It needs to be checked whether this change also works logged-in and downloads the same quality as before. If not, some of the changes need to be encapsulated in a check whether we are logged-in or anonymously.

aandergr · 2025-11-08T17:26:56Z

instaloader/structures.py

+    @classmethod
+    def from_new_iphone_struct(cls, context: InstaloaderContext, media: Dict[str, Any]):
+        """Create a post from a given new_iphone_struct.
+
+        .. versionadded:: 4.15"""


This classmethod is actually redundant; it seems the Post constructor can be invoked instead.

aandergr · 2025-11-08T17:27:14Z

instaloader/structures.py

                for node in media["carousel_media"]]}
        return cls(context, fake_node, Profile.from_iphone_struct(context, media["user"]) if "user" in media else None)
-
+    


Unrelated change; please remove.

aandergr · 2025-11-08T17:28:01Z

instaloader/structures.py

-                                          video_url=node['video_url'] if is_video else None)
+                                          video_url=str(node['video_url']) if is_video else "")


This triggers a broken sidecar node video downloading for every non-video sidecar node.

aandergr · 2025-11-08T17:28:22Z

instaloader/structures.py

+                metadata = self._context.get_iphone_json('api/v1/users/web_profile_info/',
+                                                         params={'username': self.username})


Is this change relevant?

aandergr · 2025-11-08T17:30:48Z

.gitignore

+venv/
+*.egg-info/
+build/


This change is unrelated; I suggest to add those to your local .git/info/exclude to not have them listed as untracked files. (also see https://git-scm.com/docs/gitignore )

billbeans · 2025-11-08T17:32:26Z

Yes, I mentioned in my earlier comment that this branch can't download reels, but what I really meant was highlights - I had the terminology wrong. I find that using this branch for anonymous fetching of posts then logging in on the main branch to grab highlights is a good strategy, as it greatly reduces the amount of requests you're making on your account, lowering the chances of a ban. Of course, you don't get the full resolution content, but I think it's close enough.

Also, fix mypy error regarding `video_url` in the `shortcode_to_mediaid` function

…ruct` for legacy reasons Also, add tests for shortcode retrieved posts

…ata`

aandergr · 2025-11-08T21:01:04Z

We did some major cleanup on this PR: We removed the Post.from_new_iphone_struct. It was only used in Instaloader.get_location_posts and Hashtag.get_top_posts, where it did not fix the download, neither anonymously nor logged-in, and in Hashtag.get_posts, which is deprecated.

So the only significant change was in Profile.get_posts, which fixed the download for the anonymous case, but in return broke it for the logged-in case. We rewrote this change to affect the behavior only when not logged in.

The change in the unit test did not test any of the other changes.

SamuMazzi · 2025-11-10T08:54:56Z

Sorry for the late reply @aandergr, but thanks for the support and all those fixes :)

kakaroto mentioned this pull request Mar 15, 2025

Highlights cannot be downloaded #2535

Closed

SamuMazzi mentioned this pull request Mar 28, 2025

Error 401 - Unauthorized "fail" status, message "Please wait a few minutes before you try again." #2511

Open

Arxtage mentioned this pull request Apr 1, 2025

Fix reels download #2545

Closed

Arxtage approved these changes Apr 1, 2025

View reviewed changes

SamuMazzi force-pushed the master branch from 9b3ccc2 to 87cfbf7 Compare August 26, 2025 22:29

aandergr changed the base branch from master to upcoming/v4.15 November 8, 2025 17:17

aandergr reviewed Nov 8, 2025

View reviewed changes

SamuMazzi and others added 5 commits November 8, 2025 20:07

Fix anonymous fetch of users and posts

b1d2c70

Also, fix mypy error regarding `video_url` in the `shortcode_to_mediaid` function

split old Post.from_iphone_struct with new `Post.from_new_iphone_st…

524f700

…ruct` for legacy reasons Also, add tests for shortcode retrieved posts

update .gitignore and put params in the right place in `_obtain_metad…

981b59d

…ata`

Remove redundant Post::from_new_iphone_struct staticmethod

8612024

Revert clearly unnecessary changes

ca4621a

aandergr force-pushed the master branch from 601272e to ca4621a Compare November 8, 2025 20:02

aandergr added 3 commits November 8, 2025 21:29

Fix Profile.get_posts

10dc545

Revert further unrelated changes

a81285d

Restore unrelated change in unittests

c5bbcc3

aandergr merged commit 6490878 into instaloader:upcoming/v4.15 Nov 8, 2025
8 checks passed

		for node in media["carousel_media"]]}
		return cls(context, fake_node, Profile.from_iphone_struct(context, media["user"]) if "user" in media else None)

		video_url=node['video_url'] if is_video else None)
		video_url=str(node['video_url']) if is_video else "")

		metadata = self._context.get_iphone_json('api/v1/users/web_profile_info/',
		params={'username': self.username})

Uh oh!

Fix anonymous fetch of users and posts #2533

Fix anonymous fetch of users and posts #2533

Uh oh!

Conversation

SamuMazzi commented Mar 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anaacaren commented Mar 15, 2025

Uh oh!

kakaroto commented Mar 15, 2025

Uh oh!

SamuMazzi commented Mar 16, 2025

Uh oh!

kakaroto commented Mar 17, 2025

Uh oh!

jaybiirds commented Mar 20, 2025

Uh oh!

xathian commented Mar 22, 2025

Uh oh!

xathian commented Mar 23, 2025

Uh oh!

Plaoo commented Mar 24, 2025

Uh oh!

latinokodi commented Mar 28, 2025

Uh oh!

SamuMazzi commented Mar 28, 2025

Uh oh!

willianbriotto commented Apr 7, 2025

Uh oh!

latinokodi commented Apr 8, 2025

Uh oh!

xcaptain commented Apr 29, 2025

Uh oh!

gabriele2605 commented May 13, 2025

Uh oh!

IgorArnaut commented May 22, 2025

Uh oh!

VolpiGiuliano commented Jun 29, 2025

Uh oh!

aandergr commented Jul 26, 2025

Uh oh!

Speedy1991 commented Aug 14, 2025

Uh oh!

SamuMazzi commented Aug 28, 2025

Uh oh!

ekalin commented Aug 28, 2025

Uh oh!

billbeans commented Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

billbeans commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aandergr commented Nov 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aandergr commented Nov 8, 2025

Uh oh!

aandergr Nov 8, 2025

Choose a reason for hiding this comment

Uh oh!

aandergr Nov 8, 2025

Choose a reason for hiding this comment

Uh oh!

aandergr Nov 8, 2025

Choose a reason for hiding this comment

Uh oh!

aandergr Nov 8, 2025

Choose a reason for hiding this comment

Uh oh!

aandergr Nov 8, 2025

Choose a reason for hiding this comment

Uh oh!

billbeans commented Nov 8, 2025

Uh oh!

aandergr commented Nov 8, 2025

Uh oh!

Uh oh!

SamuMazzi commented Nov 10, 2025

SamuMazzi commented Mar 13, 2025 •

edited

Loading

billbeans commented Sep 29, 2025 •

edited

Loading

billbeans commented Oct 30, 2025 •

edited

Loading

aandergr commented Nov 8, 2025 •

edited

Loading