The purpose of these tests was to check mixnet message delays and ensure that messages are propagated as expected. See Conclusions section for a list of properties that have been checked. This is a follow-up to previous mixnet tests: Nim-libp2p Mix - May 2025
In each experiment, mixnet nodes and gossip nodes are set up. A set of nodes are designated as publishers, and they each publish messages in order, with some delay between messages. The messages are marked in the mixnet logs as Sender, Intermediate, and Exit. After exiting the mixnet, messages are referred to just as gossip, and each node that receives the message logs when it first sees it.
For a full description of the experiment and parameters, see Experiment Setup
There are no messages being gossiped before they finished traversing the mixnet ✅
Exit nodes are the first nodes to receive messages outside the mixnet ✅
There is a huge delay for mixnet messages (see Mixnet hop times issues) ❌
This appears to be because of the multistream select handshake. There are a total of 7 outbound connections in the handshake+sending the message.
Gossip messages propagate with the expected delay ✅
All messages received by all nodes ❌
Important: We are using only 1 mix tunnel.
When we were doing experiments with mix, we run into many different scenarios that made the analysis complex. Some of those were:
Log line regarding too many connections:
Failed to send message to next hop: tid=12 err="failed new dial: failed getOutgoingSlot in internalConnect: Too many connections
After withMaxConnections(250) was added to the switch builder, this log line no longer showed up in experiments. See Log errors issues below.
Log line regarding internal connections:
Failed to dial next hop: tid=12 err="failed new dial: Unable to establish outgoing link in internalConnect"
We let the ACZ team know this. It looks like it can be related to the reply flow or when the packet is being sent to the first hop. Currently investigating by the team.
Insane values for latencies. We encountered latencies over 900 seconds.
Warning log lines about “no destination read behavior”
no destination read behavior for codec tid=12 codec=/meshsub/1.2.0
In order to pin-point those issues, we had to play with the ratio of mix-gossipsub nodes, number of messages, and artificial delays.