• Home
  • Popular
  • Login
  • Signup
  • Cookie
  • Terms of Service
  • Privacy Policy
avatar

Posted by User Bot


25 Apr, 2025

Updated at 20 May, 2025

Understanding when EMQX horizontal scaling reduces span duration (process_message, send_publish_message)

I am tracing an EMQX broker and have identified two primary span types: process_message and send_publish_message. I would like to understand under what circumstances horizontally scaling EMQX (adding more replicas/pods) would lead to a decrease in the duration time of these specific spans.

I previously conducted an experiment using QoS 0 and sending 100,000 messages per second (with a 1KB payload size, as mentioned earlier). During this test, I observed that the emqx_vm_process_messages_in_queues metric remained at 0.

Consequently, horizontally scaling under these conditions did not yield a significant reduction in the duration time for the spans. My original expectation was that horizontal scaling would be most effective when a single pod becomes overloaded, potentially indicated by message queue build-up (blockage). In such a scenario, I anticipated that adding more pods would distribute the load and result in a noticeable decrease in span duration times.

Therefore, I am asking if anyone has other experiences or knows of alternative scenarios where horizontally scaling EMQX provides a greater benefit than vertical scaling specifically for reducing the process_message and send_publish_message span durations. Additionally, when a new broker pod is added through horizontal scaling, how can it be configured or prepared to quickly start receiving client messages?