Crawling does not fetch all images of srcset

[ADevAtAtTime]

I am using browsertrix-crawler:1.7.0 in a docker container.
On my local windows machine I have no problems and all images are fetched,
but when I try to crawl on a onpremise machine only images fitting the breakpoint for the default windowSize 1360,1020 are fetched.

The images per breakpoint are defined in a srcset.

Do you have a clue, why the behaviour deviates and know a fix, so that all images are crawled?

[ikreymer]

1.7.0 is a fairly old version, we're on 1.11.x now, so would definitely recommend trying with latest.

If that doesn't work, there's not much we can do without looking at the specific example. The Autofetch behavior should be getting all of the <srcset> URLs if you're running with the default behaviors enabled.

If you'd like to share the URL here or send it to us via URL, we can try to reproduce.

[ADevAtAtTime]

I am still on 1.7.0 because fetching images stopped working altogether starting with 1.8.0. The issue ist still present in 1.11.0 (I checked it, recently). It seems like a change in pupeteer caused this.

I also have to correct myself. Its not an onpremise-machine but a docker container in a kubernetes cluster in a computer center.

I get errors like:
"error":{"name":"ProtocolError","message":"Protocol error (Fetch.takeResponseBodyAsStream): Invalid InterceptionId."}

[ikreymer]

I am still on 1.7.0 because fetching images stopped working altogether starting with 1.8.0. The issue ist still present in 1.11.0 (I checked it, recently). It seems like a change in pupeteer caused this.

I also have to correct myself. Its not an onpremise-machine but a docker container in a kubernetes cluster in a computer center.

Are you running the full Browsertrix app then? Or what Kubernetes cluster - curious about your deployment!

I get errors like: "error":{"name":"ProtocolError","message":"Protocol error (Fetch.takeResponseBodyAsStream): Invalid InterceptionId."}

I think this will finally be fixed with #975 actually, it was a bug on our end that caused these errors to start popping up.

Hopefully the next 1.12.0 beta will fix this issue fully!