HackerOne API Rate Limiting Issue

[Nishantbhagat57]

Hi @JoshuaMart,

Hope you're doing well :)

Recently, I've been consistently getting rate limited by the HackerOne API. I'm using ScopesExtractor in combination with bbscope and another tool of mine to collect assets, and it's possible that both tools are calling the HackerOne API at the same time.

It was working flawlessly before, but this issue started only recently. I'm now consistently running into rate limits, so I suspect there may have been some changes to HackerOne's API rate limiting.

[2026-06-22 18:34:59] ERROR [HackerOne] Failed to fetch/parse program xyz: Failed to fetch scopes for xyz: HTTP 429
[2026-06-22 18:34:59] WARN GET https://api.hackerone.com/v1/hackers/programs/xyz_bbp/structured_scopes?page%5Bnumber%5D=1&page%5Bsize%5D=100 → 429 (0.19s)

For now, I think the best approach would be to implement a wait-and-retry mechanism similar to ScopesExtractor’s handling of Discord rate limits. Allowing up to 5 retries (or more if needed), with exponential backoff between attempts, seems reasonable. It's better to wait a bit and fetch the data reliably than to skip it and wait until the next cycle, which could be 2-3 hours later.

Let me know if you need any additional logs or details from my side.

As always, thanks for maintaining such a useful project ❤️

Thanks and Regards,
Nishant

[Nishantbhagat57]

Hi @JoshuaMart,

I'm still consistently getting rate limited, even when only a single instance of ScopesExtractor is running and nothing else is using the HackerOne API. This makes me think HackerOne may have changed their API rate limits. I'll try to validate that later when I'm back at my laptop.

Another thing I noticed: if ScopesExtractor is unable to fetch a program's assets, it seems to mark the program as removed. I don't think that's the correct behavior, since a temporary API failure or rate limit doesn't necessarily mean the program no longer exists. As a result, the existing program data gets removed from the database. I think it would be better to keep the existing data unless it's explicitly confirmed that the program or its assets have actually been removed.

Looking forward to your review.

Kind regards,
Nishant

[JoshuaMart]

Hi @Nishantbhagat57,
I just pushed two PRs (which I haven't tested in production myself)

I'm using ScopesExtractor in combination with bbscope and another tool of mine to collect assets

Actually, there’s no doubt that the rate limit issues could stem from this so may I ask why you’re using three different tools ?

To be honest, it’s already a pain to keep up with every change on the platforms, especially since I haven’t been hunting on H1 or Bugcrowd for a while now, so this is the kind of edge case I’d rather not have to deal with.

---

More generally, rate limiting (across all platforms) and issues related to program fetching and scopes should now be taken into account

Regards,
Joshua

[Nishantbhagat57]

Hi @JoshuaMart,

Thanks for the update.

I just tested the latest changes and they work. From what I've seen, the rate limit usually only resets around the 5th retry, so to make it a bit more reliable I've increased the maximum retries to 10 instead of 5.

Actually, there’s no doubt that the rate limit issues could stem from this so may I ask why you’re using three different tools ?

There are a couple of edge cases that ScopesExtractor doesn't currently cover, so some assets may not get saved in the database. It's probably less than 1%, but for my use case I prefer not to miss any assets at all, so I also run bbscope alongside it as an additional source.

That said, the rate limiting doesn't seem to be caused by running both tools at the same time. I confirmed this by running only a single instance of ScopesExtractor while making sure nothing else was using the HackerOne API, and I was still getting rate limited consistently. I think HackerOne has probably made their API rate limits a bit more aggressive recently.

The new exponential backoff retry logic definitely works though.

There was also another issue regarding SQLite3::BusyException: database is locked that I ran into. I fixed it locally (with some help from Claude), tested it for a while, and it seems to be working fine. I'll open a PR for that shortly.

Thanks again for maintaining this project. I really appreciate you taking the time to address these issues. I know it's not easy keeping up with all the platform changes when you're not actively hunting on h1 or bc anymore.

I think with these recent fixes ScopesExtractor has become really stable, and while debugging them I've become fairly familiar with parts of the codebase as well. Since I depend on this project quite a bit, I'll be happy to help out with future fixes or open PRs whenever I come across something that needs attention.

Thanks again :)

[JoshuaMart]

There are a couple of edge cases that ScopesExtractor doesn't currently cover, so some assets may not get saved in the database. It's probably less than 1%, but for my use case I prefer not to miss any assets at all, so I also run bbscope alongside it as an additional source.

Couldn't another tool (bbscope or your custom script) just cover all your needs ?

There was also another issue regarding SQLite3::BusyException: database is locked that I ran into. I fixed it locally (with some help from Claude), tested it for a while, and it seems to be working fine. I'll open a PR for that shortly.

Thanks for the PR. I made a small change, but it was perfect, it's merged.

Regards,
Joshua

[Nishantbhagat57]

Hi @JoshuaMart,

Compared to bbscope, I think ScopesExtractor does a much better job of normalizing scope inconsistencies. There are just a few edge cases (probably well under 1%) where an asset gets skipped because it isn't currently handled by a validator. I've only noticed this on a like 2-3 h1 and bugcrowd programs. For example, something like *.staging*.xyz.com may be ignored. I understand these are rare edge cases and probably aren't worth adding special handling, but for my workflow I'd rather not miss even a single asset, since that could potentially mean missing a valid vulnerability and bounty opportunity.

Another reason I was using bbscope alongside ScopesExtractor was that, before your recent fix, ScopesExtractor would remove a program's data whenever an error occurred while fetching it. Now that you've addressed that issue, it's working much better, and I'm a lot more comfortable relying on it.

As for the custom script, it's only a small helper that interacts with other HackerOne API endpoints (for example, pulling reports, hacktivity).

Thanks again for all the recent fixes and improvements. The updates are working well :)

Regards,
Nishant