Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

When retrieving a user's following list, an error occurs with the cursor when the count reaches approximately 2300. #34

Open
0xfocu5 opened this issue Dec 17, 2024 · 1 comment

Comments

@0xfocu5
Copy link

0xfocu5 commented Dec 17, 2024

Here is test code。
when the count reaches approximately 2300, The cursor will repeatedly fetch data from the beginning of the list.

cursor := ""
userId:="1323413840235696130"
num := 0
for {
		users, cursor, err = scarper.FetchFollowingByUserID(userId, 200, cursor)
		if err != nil {
			return err
		}
		num += 50
		fmt.Println(num)
	}
@cmj
Copy link

cmj commented Dec 19, 2024

At a glance, it seems the line here is just sending an empty string, which would make the requests start over.

https://github.com/imperatrona/twitter-scraper/blob/152a0a1c2c60df3812b3fd9cfcaf61905f68a80f/follows.go#L75C16-L75C18

(I'm not sure how to fix this, but worth looking into)

Also, that endpoint isn't replying with 200 results anymore; only 50? So you're seeing rate limits much sooner.

I'm still getting 200 off the old v1.1 endpoint https://api.twitter.com/1.1/friends/list.json?count=200 which results in about 26.8K before the limits are hit. People following that many are rare, but was just curious.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants