Hi, everyone.? The folks at InfoQ have published an article that I've written about the new consistent hashing technique that is being added to Wallaroo.
The 0.5.0 Wallaroo release added support for "dynamic keys" [1].? Wallaroo can now automatically recognize new keys and route data to the appropriate partitioned state step.? The routing algorithm used today (including the recent 0.5.2 release) is very general and is unaware of load balancing considerations.? We are now integrating a technique called Random Slicing into Wallaroo to add very fine-grained control over the distribution of keys across Wallaroo worker processes.
The second half of the InfoQ article [2] describes the Random Slicing consistent hashing technique, together with several illustrations to show it adapts to changing cluster membership and load balancing criteria.? The first half of the article describes some earlier consistent hashing techniques, including the one used by Amazon's original Dynamo database and adopted by Riak and other distributed databases.
John Mumm, Andy Turley, and I are excited to bring enhancements to Wallaroo's dynamic keys, load balancing, and crash resilience features to Wallaroo in the next month.? If you have questions about any of this work, please feel free to contact me by email.
Scott Lystig Fritchie