Reqwest slower than Go in GKE Autopilot cluster

I'm having trouble troubleshooting slow performance with reqwest when running in a Google Kubernetes Engine Autopilot cluster. The exact same code seems to run fine everywhere (even on a Google Compute Engine instance) but the second it is in a GKE Autopilot cluster it is about half as fast as Go.

I'm only testing using http1.1 - I've disabled h2 in my tests.

Here is what I've been testing with: GitHub - bennetthardwick/http-download-testing.

I've put them up as a container - you can test them out like this:

docker run -it bennetthardwick/http-download-testing:ubuntu-24.04

export TEST_URL='some url'

make bench_go # run go | pv > /dev/null
make bench_curl # run curl | pv > /dev/null 
make bench_reqwest # run reqwest | pv > /dev/null 

This is the abridged Go code that I've written:

client := http.Client {
	Transport: &http.Transport {
		TLSNextProto: make(map[string]func(authority string, c *tls.Conn) http.RoundTripper),
	}
}

res, err := http.Get(".. url ..")

io.Copy(os.Stdout, res.Body)

This is the abridged Rust/reqwest version that I've written:

let mut stdout = std::io::stdout().lock();
let client = request::ClientBuilder::new().http1_only().build().unwrap();

let mut bytes = client.get(".. url ..").send().await.unwrap().bytes_stream();

while let Some(next) = bytes.next().await {
	stdout.write_all(&next.unwrap()).unwrap();
}

Running in us-central1 everything is between 200-300MiB/s. But reqwest drops down to half when running in GKE.

Here are the results I see:

GCP instance, g1-small, GCS object same region, https

GCP GKE Autopilot Pod, n1 machine class, GCS object same region, https

One thing to note is Ureq doesn't seem to suffer from the slowdown - which starts to feel like it's some kind of tokio related issue. I've tried changing a bunch of settings and nothing helps. If anyone has any ideas I'd really appreciate it!

1 post - 1 participant

Read full topic