External DNS does not delete records using Openstack Designate

[frittenlab]

I already found a similar issue but the request was not answered. So I will open another issue:

#751

Adding A-Records with external dns and openstack designate works fine. I have set the policy to sync. On AWS the A-Records get removed when I delete the Load Balancer Service. Using Designate the A-Record never gets removed. The worst thing is you cannot recreate a Service with the same domain name until you manually delete the A-Record from the DNS zone. The openstack API sends a Duplicate Recordset error:

time="2019-07-30T13:13:13Z" level=info msg="Creating records: nginx.example.com./A: 185.X.X.X" time="2019-07-30T13:13:13Z" level=error msg="Expected HTTP response code [201 202] when accessing [POST https://designate.cloud.example.com:9001/v2/zones/e86a4b09-70aa-41ff-abcd-7bba6c8c820e/recordsets], but got 409 instead\n{\"message\": \"Duplicate RecordSet\", \"code\": 409, \"type\": \"duplicate_recordset\", \"request_id\": \"req-c7588b3f-aaeb-4769-85b7-b6f32cc23325\"}"

After deleting the A-Record manually I can successfully add a new one:

time="2019-07-30T13:18:13Z" level=info msg="Creating records: nginx.example.com./A: 185.X.X.X" time="2019-07-30T13:18:13Z" level=info msg="Creating records: nginx.example.com./TXT: \"heritage=external-dns,external-dns/owner=default,external-dns/resource=service/default/nginx\"

[Benvandamme]

Same problem here. Openstack Queens with external-dns v0.5.16-7. Kubernetes deployed via kubespray.

[stefanandres]

@Benvandamme You might try #1136 - this will make updating records work for now.

[mcayland]

I've just hit this exact issue today - are there any plans to merge the above PR in the near future? Or is it necessary to build from source and manually patch for now?

[Benvandamme]

I built it from this source https://github.com/syseleven/external-dns/tree/txtregistry-keep-endpoint-labels it makes it better but the txt-records still trigger the duplicate-error. As workaround i disabled them and created a dns-zone only for this kubernetes-cluster. This seems to work.

Thank you for your help.

[mcayland]

I've been reading over the discussions on #1136 and from there and what you say above about TXT records not working, it feels like this PR is still missing something.

@njuettner - as you were reviewer for that patchset, can you explain a bit more about how this should work? It seems reasonable to assume that this feature worked when it was first merged, and so something or some underlying assumption must have changed within the past year or so to cause this to break.

[multi-io]

I've been reading over the discussions on #1136 and from there and what you say above about TXT records not working, it feels like this PR is still missing something.

I'm the author of that PR. Yes, the failed TXT record update is why I wrote that the designate provider is only "half working" with it. I don't fully understand what it's doing or why it even wants to update the TXT record. The whole "remember stuff in labels" approach seems weird; no other provider does this. We'll probably look into this some more in the future and write a real fix, but this PR was a quick and easy improvement.

[mcayland]

I see, thanks. AFAICT the TXT record is really a way of (ab)using designate so that all of the metadata related to the DNS record can also be stored within OpenStack itself - or at least I don't see an obvious way of adding metatdata tags and/or properties to individual records, which is something that can be done with most other OpenStack objects.

It feels as if the TXT and A records should be combined into a single A record + metadata object before processing which I believe is what your patch is doing?

@Benvandamme you mentioned that you've disabled TXT records for now in order to get this to work. Can you explain how you did this?

[Benvandamme]

HI @mcayland i added following parameter to the executable and built my own container with the needed authentificationparameters for my cluster hardcoded.

--registry=noop

[mcayland]

@multi-io whilst searching for related information I discovered an old PR at #859 which appears to be an attempt to properly fix the underlying issue of using labels to store some metadata.

Unfortunately it was never merged due to a bug discovered in testing, and I don't see any recent updates either. Perhaps a better approach might be to try and fix the outstanding issue in the above PR?

[mcayland]

After spending most of today playing and testing with the source code, it seems to me that the txt registry has some fairly obvious flaws in it.

There is certainly a source of confusion as to whether the txt registry Records() function should return 1 EndPoint for each target, or 1 EndPoint of each of the A and TXT records. From what I can tell the txt registry is the only one that drops the TXT EndPoint before supplying EndPoints to the plan - compare with the aws_sd_registry which behaves more as I would expect.

Since the TXT EndPoint (along with its associated labels and provider-specific data have now disappeared) there must be some way of regenerating these records somewhere so they can be deleted by the provider. And that magic happens in ApplyChanges() where the remaining A record entry is "cloned" into a TXT record so it can be scheduled for deletion along with the A record.

However this still fails with the designate provider because the A records and TXT records have different designateRecordSetIDs; since the TXT records has been cloned from the A record, not only have we lost the id of the recordset for the TXT record, we end up trying to delete the same recordset twice which inevitably ends in failure.

In short, the fix for this is going to require some thought around what the interface contract expects from the registry implementations. Currently my feeling is that similar to the aws_sd_registry the txt registry should not be filtering TXT records, but should instead allow them to be returned so that they are accessible from ApplyChanges().

Unfortunately I'm about to go on holiday for a few weeks which means I won't have any time to look at this for a while, but I thought it was worth posting this analysis in the hope that one of the project developers can confirm how the txt registry should be behaving, and even better if someone else can take this and make some further progress in the meantime.

[mcayland]

After some overnight inspiration I realised that as long as the txt registry was able to handle ProviderSpecificProperty then it would be possible to make the designate provider work correctly with only minimal changes.

As a result I now have a branch that "works for me" in that I can successfully remove and recreate an EndPoint and associated DNS A/TXT records automatically via designate.

The branch is available on github at https://github.com/mcayland/external-dns/commits/designate/fix-delete-TXT-records for people to test - if it looks good, let me know and I'll raise a PR and see if I can get it merged.

[mcayland]

Just tickling this issue again now that I'm back - has anyone tried the branch above to see if it works for them?

[larsweiler]

Thanks for your branch. I tried that in a cluster with four ingress resources. In the end the TXT records in designate were deleted after some time and only the A records remained. So in short, works for me. But this is not a thorough testing with all situations which might occur.

[mcayland]

@larsweiler thanks for the feedback! I've been away from this particular project again for a while, so I'm guessing from your comment above that this is still broken? I'll see if I can find a few moments to test master again, and if that is still failing will raise a proper PR.