The current implementation of the synchronization service uses Redis and is implemented within the SDK itself. It is extensively inspired by Apache ZooKeeper and Apache Curator recipes. At the time we needed to have Testground ready quickly and robust so we "outsourced" the sync primitives to Redis. More about the current implementation here.

Problem

We have been working on developing a JavaScript SDK for Testground that must work on both Node.js and browsers. Redis is not officially supported on the browser, thus we could not use the native library to connect from JavaScript to Redis. We considered Webdis, an HTTP proxy for Redis. However, it does not support MULTI and EXEC, which are essential for the current implementation.

Solution

The solution we came up with is to implement our own independent Synchronization Service that will no longer be embedded in the SDKs. This sync service will be run in parallel with the tests, and each instance can start a WebSocket connection to the sync service and work through there.

With this new service, we also decided to remove Redis. Redis is currently the central piece of our sync service. However, we only have 4 synchronization directives that can be easily implemented in Go without requiring a third party service:

• Barrier
• Signal Entry (to the barrier)
• Publish
• Subscribe

Advantages vs Disadvantages

The main advantage of replacing the current implementation by an independent sync service is that it will allow to simplify the SDKs' logic and facilitate the creation of SDKs in new languages in the future. Besides, fixing and adding new features is also much easier. We will also be able to run tests in more environments, such as browsers, since WebSockets run over HTTP.

On the other hand, we will have one more moving piece to maintain. And that's where removing Redis comes up too. Since we're moving to an independent sync service, we also have the opportunity to implement the few directives ourselves since they are quite simple.

When the sync service was created, the decision to use Redis also passed by the fact that it is performant and it would allow for creativity on the developer end to build new sync directives. More than a year has passed, and the initial directives that we created are still the ones in use. Thus, there is not a lot of advantage on keeping Redis.

Plan

The new sync service is being created on the following repository: testground/sync-service.

We plan for a smooth transition, where upgrading SDKs will be enough to start using the new sync service.

We want to keep this decision transparent and open for feedback. Please let us know if you have any objections and concerns about this change. We will test the new sync service to ensure that there's no performance regression and that it is working as it should.