dnsexit2: Update multiple hosts at a time when possible #684

rhansen · 2024-06-03T07:32:30Z

TODO:

Update existing tests.
Add new test cases for multiple hosts.
Test with DNSExit. Thank you @jortkoopmans!
Wait for DNSExit to fix their service (see the comment below).
Test again with DNSExit.

jortkoopmans · 2024-06-08T22:00:14Z

Sorry for the delay, I've been having some challenges on testing this. While testing this with multiple hosts, the produced JSON data is looking perfectly fine to me, but I'm receiving this response;

{"code":6,"message":"System Error - java.lang.IndexOutOfBoundsException: Index: 2, Size: 2"}

Interestingly, when only testing with a single host, it works OK.

Finally, I've decided to contact DNSExit support (just did this), in hopes of moving this forward.
I'll keep you posted of course.

However, one other thing I've noticed, is that the cache is behaving strangely. While the cache is correctly updated to have for each host;

status=failed,status-ipv4=failed,status-ipv6=failed,

it also has the IPs set to it, and the next run will happily print: skipped: IPv4 address was already set to [..]
See nic_updateable around line 3755.
I haven't looked into understanding the process for writing the cache and workings of nic_updateable, but I would expect it would run when update is failed (and do something with the backoff timers etc).

rhansen · 2024-06-13T03:22:39Z

Thank you for testing @jortkoopmans!

{"code":6,"message":"System Error - java.lang.IndexOutOfBoundsException: Index: 2, Size: 2"}

This definitely looks like a server-side bug at DNSExit. I'll put this PR on the back burner until I hear that the issue has been fixed.

However, one other thing I've noticed, is that the cache is behaving strangely.

That's an old bug in ddclient that arises with use=ip, usev4=ipv4, or usev6=ipv6. If you change the use* setting to something else that issue goes away.

jortkoopmans · 2024-06-24T11:35:49Z

Just a brief update, I've not received word from DNSExit support yet. Today I've sent a friendly reminder.

jortkoopmans · 2024-10-07T21:48:16Z

Apologies for the extremely late response @rhansen.

After some attempts to get a response on my support request from DNSExit, I finally caught their attention, and they were friendly enough to look into the issue and subsequently reported they fixed the issue on their end.

It then sat on my desk for a bit 😬, but I finally got around to testing it again based on this branch.
With good results; the multiple hosts feature works fine now 🥳.

I then briefly looked into resolving conflicts on this old branch with master, but hit a snag with reworks on the group_hosts_by subroutine and some novel ways of verifying the config. My hope is that you can resolve this more swiftly 🤞 (as you're the author of the majority of these changes).

This makes the tests easier to read and extend.

rhansen · 2025-01-10T00:47:58Z

@jortkoopmans @DiSHTiX Would one (or both) of you please test this one final time?

indrajitr · 2025-01-10T05:26:11Z

I signed up with dnsexit and grabbed a free domain to test this.

Here are my observations:

Case 1 (fail):

ddclient.conf:

usev4=webv4, \
protocol=dnsexit2, \
zone=linkpc.net, \
password='key' \
foo.linkpc.net,bar.linkpc.net

ddclient.cache:

## ddclient-4.0.0-rc.3
## last updated at Thu Jan  9 22:59:45 2025 (1736485185)
atime=1736485185,status-ipv4=failed foo.linkpc.net
atime=1736485185,status-ipv4=failed bar.linkpc.net

Case 2 (pass):

ddclient.conf:

usev4=webv4, \
protocol=dnsexit2, \
password='key' \
foo.linkpc.net,bar.linkpc.net

ddclient.cache:

## ddclient-4.0.0-rc.3
## last updated at Thu Jan  9 23:01:30 2025 (1736485290)
atime=1736485290,ipv4=208.x.y.z,mtime=1736485290,status-ipv4=good foo.linkpc.net
atime=1736485290,ipv4=208.x.y.z,mtime=1736485290,status-ipv4=good bar.linkpc.net

Case 3 (pass):

ddclient.conf:

usev4=webv4, \
protocol=dnsexit2, \
zone=linkpc.net, \
password='key' \
foo,bar

ddclient.cache:

## ddclient-4.0.0-rc.3
## last updated at Thu Jan  9 23:03:33 2025 (1736485413)
atime=1736485413,host=foo.linkpc.net,ipv4=208.x.y.z,mtime=1736485413,status-ipv4=good foo.linkpc.net
atime=1736485413,host=bar.linkpc.net,ipv4=208.x.y.z,mtime=1736485413,status-ipv4=good bar.linkpc.net

jortkoopmans · 2025-01-10T09:48:11Z

@indrajitr ; thanks for testing, one thing I do recall is that behavior on non-existing records can be unpredictable. For your case 1 (fail); did you have the foo and bar records already existing?

@rhansen ; I will try to also run a test soon!

indrajitr · 2025-01-10T22:07:03Z

@jortkoopmans, more details:

Please ignore case 3, that's not a 'pass', that errors out with "hostname does not end with the zone". Sorry about that.
case 1 indeed fails consistently. I have tried setting up IP address with case 2 config and come base to try case 1. It results in failure as well.

Enabled debug for more details, I see the API call is using following JSON payload:

{
    "update": [
        {
            "content": "123.x.y.z",
            "ttl": 5,
            "name": "bar",
            "type": "A"
        },
        {
            "type": "A",
            "name": "foo",
            "ttl": 5,
            "content": "123.x.y.z"
        }
    ],
    "apikey": "secret",
    "domain": "linkpc.net"
}

and the POST fails with following response:

DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request]> CURL: /usr/bin/curl
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request]> CURL Tempfile: /tmp/ddclient_RcPq8oRqbh
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request][RESPONSE]> reply:
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request][RESPONSE]  HTTP/1.1 200 200
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request][RESPONSE]  Date: Fri, 10 Jan 2025 21:41:26 GMT
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request][RESPONSE]  Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_jk/1.2.48
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request][RESPONSE]  Set-Cookie: JSESSIONID=897AA476C44FD6F534AE5FE238399D6A; Path=/; Secure; HttpOnly
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request][RESPONSE]  Content-Length: 69
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request][RESPONSE]  Content-Type: application/json;charset=UTF-8
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request][RESPONSE]  
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request][RESPONSE]  
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request][RESPONSE]  
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request][RESPONSE]  
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request][RESPONSE]  
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request][RESPONSE]  {"code":2,"message":"API Key Authentication Error"}
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request][RESPONSE]  
DEBUG:   [dnsexit2][ARRAY(0x563b25a918c0)][HTTP request][RESPONSE]              
INFO:    [dnsexit2][ARRAY(0x563b25a918c0)]> badauth: API Key Authentication Error. The API Key is missing or wrong.
INFO:    [dnsexit2][ARRAY(0x563b25a918c0)]> server message: API Key Authentication Error
INFO:    [dnsexit2][ARRAY(0x563b25a918c0)]> server details: no details received
FAILED:  [dnsexit2][ARRAY(0x563b25a918c0)]> API Key Authentication Error. The API Key is missing or wrong.
FAILED:  [dnsexit2][ARRAY(0x563b25a918c0)]> server response: API Key Authentication Error

For case 2 there are two separate calls with the following payloads:

{
    "apikey": "secret",
    "domain": "foo.linkpc.net",
    "update": [
        {
            "name": "",
            "content": "123.x.y.z",
            "ttl": 5,
            "type": "A"
        }
    ]
}

and

{
    "apikey": "secret",
    "domain": "foo.linkpc.net",
    "update": [
        {
            "name": "",
            "content": "123.x.y.z",
            "ttl": 5,
            "type": "A"
        }
    ]
}

the POST succeeds with the following response (pasted for one host, but the other host succeeds with similar response):

DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request]> CURL: /usr/bin/curl
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request]> CURL Tempfile: /tmp/ddclient_90waZwuPuk
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request][RESPONSE]> reply:
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request][RESPONSE]  HTTP/1.1 200 200
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request][RESPONSE]  Date: Fri, 10 Jan 2025 21:40:38 GMT
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request][RESPONSE]  Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_jk/1.2.48
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request][RESPONSE]  Set-Cookie: JSESSIONID=E1E9C81C3A46689C11089FB4751E451C; Path=/; Secure; HttpOnly
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request][RESPONSE]  Content-Length: 136
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request][RESPONSE]  Content-Type: application/json;charset=UTF-8
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request][RESPONSE]  
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request][RESPONSE]  
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request][RESPONSE]  
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request][RESPONSE]  
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request][RESPONSE]  
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request][RESPONSE]  {"code":0,"details":["UPDATE Record A foo.linkpc.net. TTL(hh:mm) 00:05  IP   123.x.y.z"],"message":"Success"}
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request][RESPONSE]  
DEBUG:   [dnsexit2][ARRAY(0x55aa70fa19a8)][HTTP request][RESPONSE]              
INFO:    [dnsexit2][ARRAY(0x55aa70fa19a8)]> good: Success! Actions got executed successfully.
INFO:    [dnsexit2][ARRAY(0x55aa70fa19a8)]> server message: Success
INFO:    [dnsexit2][ARRAY(0x55aa70fa19a8)]> server details: UPDATE Record A foo.linkpc.net. TTL(hh:mm) 00:05  IP   123.x.y.z
SUCCESS: [dnsexit2][ARRAY(0x55aa70fa19a8)]> Success! Actions got executed successfully.
SUCCESS: [dnsexit2][ARRAY(0x55aa70fa19a8)]> updated IPv4 address to 123.x.y.z

This increases overhead when the verbose or debug option is disabled, but makes it possible for tests to intercept debug, info, and fatal messages.

rhansen · 2025-01-11T00:39:29Z

I pushed some logging fixes and test improvements (no other behavior changes).

@indrajitr IIUC, foo.linkpc.net and bar.linkpc.net are two different zones (maybe even separate accounts?), which is why zone=linkpc.net fails. For case 1 I think you actually want to test something like this:

usev4=webv4, \
protocol=dnsexit2, \
zone=foo.linkpc.net, \
password='key' \
host1.foo.linkpc.net,host2.foo.linkpc.net

assuming DNSExit supports third-level host names for their free second-level domain names.

indrajitr · 2025-01-11T02:31:31Z

@rhansen: foo.linkpc.net and bar.linkpc.net are actual hosts with A records available under linkpc.net which is a free domain that dnsexit allowed me to pick up.

rhansen · 2025-01-11T04:07:49Z

foo.linkpc.net and bar.linkpc.net are hostnames, but each is also a "domain" according to their API (what ddclient calls "zone"). linkpc.net is a domain in the colloquial sense, but it is not a DNSExit API "domain" which is why case 1 failed.

foo.linkpc.net and bar.linkpc.net were available for you to claim, not linkpc.net as a whole. I'm sure other users have claimed other subdomains in *.linkpc.net.

indrajitr · 2025-01-11T07:20:39Z

@rhansen you are 100% right!

A records for foo.linkpc.net, bar.linkpc.net etc. cannot be simultaneously updated using DNSExit rest API - even when they belong to the same account and have the same API Key (which is what I have in the test account I created). They are all treated like apex -ish domain (and have A records) and thus are zones in ddclient world.

assuming DNSExit supports third-level host names for their free second-level domain names

Just found out that this is actually allowed in their DNS control panel. So technically, I can have foo.linkpc.net, h1.foo.linkpc.net, h2.foo.linkpc.net all having A record 123.x.y.z. And they can be simultaneously updated as well with this JSON payload:

{
  "update": [
    {
        "type": "A",
        "name": "",
        "ttl": 5,
        "content": "10.11.12.100"
    },
    {
      "content": "10.11.12.90",
      "ttl": 5,
      "name": "h1",
      "type": "A"
    },
    {
      "type": "A",
      "name": "h2",
      "ttl": 5,
      "content": "10.11.12.91"
    }
  ],
  "apikey": "1234567890",
  "domain": "foo.linkpc.net"
}

In fact, these h1, h2 etc. don't even need to be pre-existing DNS records and are created directly with the DNS API call (which makes sense).

So the following ddclient.conf works just fine (even when h1, h2, h3 don't exist before the DNS API call from ddclient.

usev4=webv4, \
protocol=dnsexit2, \
zone=foo.linkpc.net, \
password='secret' \
h1.foo.linkpc.net,h2.foo.linkpc.net,h3.foo.linkpc.net

DiSHTiX · 2025-01-11T09:15:44Z

Oh things are moving along fast today. I was not at my pc earlier but i see indrajitr already nailed it down.

DNSexit quite awesome how they allow full control through their api, basically all types of records can be added/removed/updated on the fly and in a single api update.

thanks both for your efforts and time

jortkoopmans · 2025-01-12T00:39:42Z

Sorry I couldn't test before this was merged, but thanks @indrajitr for doing this. Indeed the zone needs to match with the claimed subdomains (under the linkpc.net TLD), well spotted @rhansen 👍 .
Good to see it all works 🚀 , thanks so much!

rhansen force-pushed the dnsexit2 branch from f653d84 to c04cbfb Compare July 11, 2024 21:01

rhansen force-pushed the dnsexit2 branch from c04cbfb to 4f6249d Compare December 19, 2024 08:22

jortkoopmans mentioned this pull request Jan 9, 2025

DNSExit: still not working since domain references FQDN instead of domain. #770

Open

rhansen force-pushed the dnsexit2 branch 3 times, most recently from 0382f13 to 2581ca9 Compare January 10, 2025 00:25

rhansen added 2 commits January 9, 2025 19:33

tests: dnsexit2 Convert tests to table-driven

3b73350

This makes the tests easier to read and extend.

tests: dnsexit2: Add test for two hosts in the same zone

b31e5e2

rhansen force-pushed the dnsexit2 branch from 2581ca9 to 4df26b1 Compare January 10, 2025 00:36

rhansen marked this pull request as ready for review January 10, 2025 00:38

rhansen requested a review from indrajitr January 10, 2025 00:48

rhansen added 5 commits January 10, 2025 19:20

logging: Move filtering and reactions to Logger class

d18b1cd

This increases overhead when the verbose or debug option is disabled, but makes it possible for tests to intercept debug, info, and fatal messages.

tests: Factor out duplicate log capture code

009033d

tests: Option to select which log messages to capture

115f23d

tests: dnsexit2: Add test for host outside of zone

3b10e37

dnsexit2: Update multiple hosts at a time when possible

63bf351

rhansen force-pushed the dnsexit2 branch from 4df26b1 to 63bf351 Compare January 11, 2025 00:33

indrajitr approved these changes Jan 11, 2025

View reviewed changes

rhansen merged commit 41170b9 into ddclient:main Jan 11, 2025
12 checks passed

rhansen deleted the dnsexit2 branch January 11, 2025 08:09

chenrui333 mentioned this pull request Jan 20, 2025

ddclient 4.0.0 Homebrew/homebrew-core#204862

Merged

dnsexit2: Update multiple hosts at a time when possible #684

dnsexit2: Update multiple hosts at a time when possible #684

Uh oh!

Conversation

rhansen commented Jun 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jortkoopmans commented Jun 8, 2024

Uh oh!

rhansen commented Jun 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jortkoopmans commented Jun 24, 2024

Uh oh!

jortkoopmans commented Oct 7, 2024

Uh oh!

rhansen commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

indrajitr commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jortkoopmans commented Jan 10, 2025

Uh oh!

indrajitr commented Jan 10, 2025

Uh oh!

rhansen commented Jan 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

indrajitr commented Jan 11, 2025

Uh oh!

rhansen commented Jan 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

indrajitr commented Jan 11, 2025

Uh oh!

Uh oh!

DiSHTiX commented Jan 11, 2025

Uh oh!

jortkoopmans commented Jan 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rhansen commented Jun 3, 2024 •

edited

Loading

rhansen commented Jun 13, 2024 •

edited

Loading

rhansen commented Jan 10, 2025 •

edited

Loading

indrajitr commented Jan 10, 2025 •

edited

Loading

rhansen commented Jan 11, 2025 •

edited

Loading

rhansen commented Jan 11, 2025 •

edited

Loading