That's an interesting use case! I'd love to learn a bit more details on how you plan to elect and denote that leader. Is this some kind of an external program that will call the service with a hardcoded machine name and essentially tell it to become a leader? What kind of service is it (any known 3rd-party software or your custom one)?

Concerns

Your proposal of adding machine-name.service-name sounds useful to me, however I have a concern that to use machine-name as part of a domain name, we need to enforce domain names to be valid DNS names. In a similar way we did this for service names before intorducing the DNS server: #53.

This is pretty straightforward and I don't see a reason to not introduce the same or identifier-like validation for names regardless of this proposal. I didn't introduce it in the first place because I was just overwhelmed by the amount of design decisions I had to make when I started the project so this one was simply ignored 😅

Another concern is that a machine name can be changed with uc machine rename. Can you think of any undesirable consequence of this? Even worse, we don't guarantee that all machine names are unique. We do optimistic checks when initialising or renaming a machine but technically it's possible to add two machines with the same name. For example, when network partitioning happens, you can rename or add a machine foo to both partitions. After the cluster heals, there will be two machines with the name foo. These all are edge cases that could be easily avoided, especially at a small scale we're likely talking about, but still I wanted to point this out.

Tradeoffs

The reason for this is the eventually consistent store without consensus (corrosion) we use. This is a tradeoff we made.
This could be addressed with distributed locks, for example, implementing Redlock (without redis). I have plans for doing this but definitely not in the near future. This will complicate the development and slow us down. For now, we can just avoid conflicts by not running uc commands modifying the same resources concurrently or when the cluster is partitioned.

For that reason, we also have 128bit unique IDs for machines which if I'm not mistaken are not visible to a user in any uc output. But internally the code operates mainly using machine IDs and only substitutes them with names when presenting them to a user.

Your proposal reminded me about the Fly.io's .internal DNS names: https://fly.io/docs/networking/private-networking/#fly-io-internal-dns which has something similar. I actually took the inspiration from it when worked on the DNS server.
I think the closest mapping of Fly.io concepts to Uncloud's is:

• Fly machine -> uncloud container
• Fly region -> uncloud ??machine

What do you think of using a machine ID instead of a name as the first iteration? How inconvenient that would be given that we can provide a way to inspect a machine to get its ID?

So the FQDN would be 2012ab97a7aff6e8bbb8bfb1883cf48a.service-name.internal or maybe we need to introduce a namespace 2012ab97a7aff6e8bbb8bfb1883cf48a.machine.service-name.internal or 2012ab97a7aff6e8bbb8bfb1883cf48a.m.service-name.internal to make it possible to introduce DNS names to resolve IPs by container IDs/names later if we want to.

Do you think we can just ignore the machine name conflicts for now and proceed with the machine-name.service-name names as you proposed?

Other ideas

This is not something I've been actively thinking about but still some food for thoughts.

I image that the uncloud cluster as an "infra platform" can potentially provide a way to help replicated services to elect leaders. Not sure how this can work, e.g. an API endpoint which service replicas can query asking "am I a leader?". And the uncloud machines will do all the quorum/leader election heavy lifting.

Somewhat similar to how leader election could be implemented in k8s: https://github.com/kubernetes/client-go/blob/master/examples/leader-election/main.go. The client leaderelection package includes some primitives you can use in your Go code to elect and check leadership. It’s implemented using etcd leases if I recall correctly. Example implementation.

Again, this is out of scope for the foreseeable future but still good to think of what an ideal solution could look like.

Apologies for the delay. Just getting back to this.

I think the idea of leader elections provided by uncloud is an interesting idea (and I could see using something like that in the future), but my initial use case is much less interesting -- I need to be able to address a specific machine's instance of a service because I have a (manually denoted) "leader". Although, "primary" might be a better term to use here than "leader".

Basically I have a process running on a single machine that updates a large file and I need to sync it to the other machines. For smaller files, I would just copy it to object storage and then download the update from each machine, but this one is large enough that I would like to rsync the changes from leader/primary machine that generates the update.

I also expect to have a few of these cases where different machines might be the "primary", so I was thinking of a simple sync service with the same functionality on each machine that I'd be able to issue commands like "B/C sync X from A", "A/C sync Y from B", etc.

There might be a better way to accomplish the above, but that's how I got to this idea of a <machine>.<service>.internal DNS entry.

Your proposal reminded me about the Fly.io's .internal DNS names

I don't recall if it was fly.io's DNS names, specifically, but the idea originated with seeing something similar in terms of internal names.

I think using the machine-id (rather than the name) makes sense and would work for me. It definitely solves a lot of issues that way with renaming and DNS-compliant names.

I see fly.io uses <machine_id>.vm.<appname>.internal and I think adding some machine or m or vm namespace in there is a good future-proofing idea in case uncloud wants to support additional naming patterns in the future (e.g. something like the region concept).

I'll update this PR to use the machine id and whichever machine/m/none option you prefer.

Also, a thought on this bit about "infra platform":

I image that the uncloud cluster as an "infra platform" can potentially provide a way to help replicated services to elect leaders. Not sure how this can work, e.g. an API endpoint which service replicas can query asking "am I a leader?". And the uncloud machines will do all the quorum/leader election heavy lifting.

While the leader election part might be a later enhancement, I could see some potential value in something like AWS's EC2 instance metadata endpoint:

• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html#instancedata-data-categories

Or maybe something like that already exists? The services' environment has HOSTNAME=<servide-id> but I could see things like the machine-id, public IP, etc also being useful for a service. Perhaps the easiest approach is to just add those to the ENV, as well, but a metadata API might make sense for more dynamic values...

And another related bit to general infra: I am running a message queue/bus service for coordination, but it's very simple usage so far. If the corrosion layer uncloud is already using could be re-used for some simple service-customizable state and/or a low-volume message bus, I could see that being very useful for many applications.

Thank you for clarifying your use case! 🙏 I think the approach you suggested with using DNS names <machine>.<service>.internal is very simple and if it works for you then great. I guess it assumes that you only have 1 replica per machine/node. Are you planning to run more than one and if so, how do you think you will distinguish containers on the same machine?

I could see some potential value in something like AWS's EC2 instance metadata endpoint:

Yeah, that's something I was actually thinking about when describing an endpoint for providing leader election info. That's definitely an interesting idea I'll keep in mind. But as you said using ENV for immutable values is likely the easiest and more versatile approach as it doesn't require the service to write any code for fetching the data.

If the corrosion layer uncloud is already using could be re-used for some simple service-customizable state and/or a low-volume message bus, I could see that being very useful for many applications.

It is indeed a great idea. Although, I'm not yet sure if it could be used as a general-purpose message bus. Corrosion is very sqlite-centric and it doesn't provide an easy way to distribute schema changes, e.g. to ensure a new table is created on all nodes. A queue could probably be emulated through a single db table we statically create on each machine. But we still need to abstract all these low level details and expose some generic interface to a user.

I'm also not yet sure if corrosion will stay long-term as we barely use its crdt capabilities and we don't really need the relational model. So for now, I'd prefer to not introduce new features that potentially make it harder to replace corrosion with something even more lightweight in the future.

Would you like me to add a new DNS entry for containers on the specific machine? Or you want to try it on your own?

Hello,

I want to connect to a service container running on a particular machine

Is the service deployed to multiple machines, but you want to specifically connect to that particular one? or you explicitly deployed it to a machine using x-machines?

Asking because if the service name is unique (there are no multiple entries), then you can just do service-name.internal and it will resolve to whatever is the machine that service is running.

Now if the service is replicated but you designated one as the leader, then you will need to find a way to share what are the available machine names so <machine-id>.<service>.internal works, specially if the leader moves machines.

Fly.io's DNS server went a bit further than just plain A/AAAA records and have TXT ones that contains the list of machines running that particular service (vms.<service-name>.internal), something like that could help with the discovery.

I also think that machine-id reference is a good compromise, but having a way to obtain the list of all machines for the specific service can also help.

Either case, it will require some coding on your end to determine which one is the leader and then connect to it. 😊

Cheers,

Fly.io's DNS server went a bit further than just plain A/AAAA records and have TXT ones that contains the list of machines running that particular service (vms..internal), something like that could help with the discovery.

DNS is simple but still powerful discovery mechanism that is easily accessible on the client without requiring to implement any custom protocols. It's not really hard to do what Fly does. Just let me know if something like TXT records with the information about service containers will be useful for you and we can easily add them.

I've just played with Fly's DNS for reference:

# dig +short TXT _apps.internal
"ps-nginx,test-nginx-ps,uncloud"
# dig +short TXT _instances.internal
"instance=d8913edc2d66e8,app=uncloud,ip=fdaa:9:8ace:a7b:1b4:2da6:5200:2,region=syd;instance=d8d3370ae66948,app=uncloud,ip=fdaa:9:8ace:a7b:4df:9cc6:43e0:2,region=syd;instance=e8254ddb603638,app=uncloud,ip=fdaa:9:8ace:a7b:2d8:2e88:fb30:2,region=syd"
# dig +short TXT vms.uncloud.internal
"d8913edc2d66e8 syd,d8d3370ae66948 syd,e8254ddb603638 syd"
# dig +short TXT regions.uncloud.internal
"syd"
# dig +short AAAA d8d3370ae66948.vm.uncloud.internal
fdaa:9:8ace:a7b:4df:9cc6:43e0:2

Would you like me to add a new DNS entry for containers on the specific machine? Or you want to try it on your own?

I'm updating (simplifying really) this MR now. I'm working on getting tests running locally, but the change will (I think) be a pretty simple addition now that it's just using the machine ID. Something like this in dns/resolver.go#updateServiceIPs:

// Add <machine-id>.m.<service-name> as a lookup
serviceNameWithMachineID := record.MachineID + ".m." + ctr.ServiceName()
newServiceIPs[serviceNameWithMachineID] = append(newServiceIPs[serviceNameWithMachineID], ip)

I guess it assumes that you only have 1 replica per machine/node. Are you planning to run more than one and if so, how do you think you will distinguish containers on the same machine?

I'm planning to just run one replica per machine, but for what I'm doing, it would be fine with multiple -- the machine-specific data is in a host volume, so any instance on the machine would be able to access it.

At the moment, I can't think of a use case for addressing a specific instance of multiple replicas on a machine that I couldn't just use the service ID DNS entry for instead.

Now if the service is replicated but you designated one as the leader, then you will need to find a way to share what are the available machine names so ..internal works, specially if the leader moves machines.

Just let me know if something like TXT records with the information about service containers will be useful for you and we can easily add them.

I imagine those will be useful for other applications (by me or others) in the future, but I should be fine with the machine-addressable service option for now.

Using the term "leader" was probably a mistake on my part. The instances of this "sync service" will be identical between replicas/machines, but they will receive commands like "sync data X from machine A". Machine A will see it's for itself and noop. Machine B will run something like rsync machine-a.m.sync.internal:/data/X /data/X.

I pushed a couple more commits:

1. I added a MACHINE ID column to the uc machine ls output.

2. I used Claude Code to make an e2e test of the internal dns lookup. It was a bit tricky (and not the fastest test), starting a service to run nslookup and pipe it to a host mounted file then execing a docker command to read that file into the test code to verify.

Not sure if there's a better way to do that test, but it was the best Claude and I could come up with so far. 😄

Nice work!

1. I added a MACHINE ID column to the uc machine ls output.

Looks good! I was also thinking of making a toggle for the ID column for both services and machines. Most of the time you don't really need to know their IDs but they add noise to the command output. For example, we can print the ID column only if --id flag is provided. Or something similar to kubectl -o wide flag that prints an extended version of the table.
I'm not asking about making any changes, just thinking out loud.

Not sure if there's a better way to do that test, but it was the best Claude and I could come up with so far. 😄

It's fine 👍. I generally try to not create new test suits that create more ucind clusters as they consume non-negligible amount of RAM and slow down tests. But it's alright, I can refactor this later if needed.

Please address the linter complains and I'm happy to merge this: https://github.com/psviderski/uncloud/actions/runs/17938964884/job/51013199831?pr=123

Please address the linter complains

I think I fixed it. golangci-lint doesn't really give much useful info on what it thinks is wrong there, and I don't use Go often enough to be familiar with the idiomatic "key: value" indentation.

(Also, I noticed that running it locally seems to complain about a few pre-existing returns in files I didn't touch with this PR. I'm guessing my local run wasn't using the exact linter config that CI runs with for some reason?)

I was also thinking of making a toggle for the ID column for both services and machines.

Yeah, that makes sense, both the toggle and adding the id to uc service ls. The table width on the machine ls was still manageable with the addition of the id, so I didn't add it there yet, but happy to look at a follow-up if you don't get to it first.