I am tracing an EMQX broker and have identified two primary span types: process_message and send_publish_message. I would like to understand under what circumstances horizontally scaling EMQX (adding more replicas/pods) would lead to a decrease in the duration time of these specific spans.
I previously conducted an experiment using QoS 0 and sending 100,000 messages per second (with a 1KB payload size, as mentioned earlier). During this test, I observed that the emqx_vm_process_messages_in_queues metric remained at 0.
Consequently, horizontally scaling under these conditions did not yield a significant reduction in the duration time for the spans. My original expectation was that horizontal scaling would be most effective when a single pod becomes overloaded, potentially indicated by message queue build-up (blockage). In such a scenario, I anticipated that adding more pods would distribute the load and result in a noticeable decrease in span duration times.
Therefore, I am asking if anyone has other experiences or knows of alternative scenarios where horizontally scaling EMQX provides a greater benefit than vertical scaling specifically for reducing the process_message and send_publish_message span durations. Additionally, when a new broker pod is added through horizontal scaling, how can it be configured or prepared to quickly start receiving client messages?