+BALUG-Admin
Al,
Thanks for noticing!
Did some DNSSEC key rollovers,
notably changing in bind 1:9.18.28-1~deb12u2
from:
auto-dnssec maintain (auto-dnssec is deprecated/obsolete and will be
going away in future)
to:
dnssec-policy
and related changes.
"Of course" starting with non-canonical less critical domains.
Yeah, if one goofs with that transition, DNSSEC can get messed up and
cause some failures (at least for a wee bit, notably depending upon
caching and relevant TTLs).
Was "only" mostly just rolling some ZSKs (and adding KSKs some), so even
there damage rather limited in scope (notably also time).
Still, there ought not be hiccups.
switching from
auto-dnssec maintain
to dnssec-policy
is a bit persnickety/hazardous (found that out many moons ago with my
very first such transition),
notably any existing keys not compatible with such policy
when one switches from
auto-dnssec maintain
to
dnssec-policy
are summarily ejected ... and that can be a (quite) bad thing.
So, pretty careful but still, not sure exactly why,
but for some zones it instantly dropped the old ZSK,
whereas for others it did the much more expected
rollover - adding the new, doing the relevant additional signings,
waiting the requisite TTL times, and only then dropping the then
vestigial old.
Not 100% sure what's made that difference between the domains.
Some are exact same gTLD and key types and sizes, and dnssec-policy and
procedures, etc. used, yet still seeing somewhat different results.
E.g. on sflug.com. vs. sf-lug.com.
See also:
https://dnsviz.net/
Can also there view the older (where saved) DNSSEC data too.
Domains that have thus far had such changes in the last day or two:
Thus far mostly worked on:
(non-canonicals and less critical):
e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa
sf-lug.comsflug.com
savingthedolph.in
sf-lug.netsflug.netdigitalwitness.orgsflug.org
And will probably hold off on these 'till all such kinks are worked out:
sf-lug.orgberkeleylug.combalug.org
Longer term bit 'o plan is to not only get all of the
auto-dnssec maintain
config bits updated to
dnssec-policy,
but also at least rotate the ZSKs if they've not been rotated in a while
(ought get rotated about monthly - and they may or may not have been
already doing that), and then work up to KSKs.
KSKs probably ought get rotated about yearly or so, but looks like most
all of 'em are 4+ years old. No extreme rush or anything, but they
could use wee bit of attention ... and also not - at least thus far -
critical. Also, if/where registrars have implemented RFC 7344 that
should make things way easier to handle that part of it.
Hmmm, and just a little while ago I blew away delegated subdomain that I'd
much earlier used for some such testing ... maybe time for some more
such testing to get to the bottom of the matter.
And if one's curious, that version of bind's default provided
configurations for that:
$ named -C | sed -ne '/^dnssec-policy "default"
{$/,/^};$/{/^$/d;p};/^dnssec-policy "insecure" {$/,/^};$/{p;/^};$/q}'
| expand
dnssec-policy "default" {
keys {
csk key-directory lifetime unlimited algorithm 13;
};
dnskey-ttl 3600;
publish-safety 3600;
retire-safety 3600;
purge-keys P90D;
signatures-jitter PT12H;
signatures-refresh P5D;
signatures-validity P14D;
signatures-validity-dnskey P14D;
max-zone-ttl 86400;
zone-propagation-delay 300;
parent-ds-ttl 86400;
parent-propagation-delay 3600;
};
dnssec-policy "insecure" {
max-zone-ttl 0;
keys { };
};
Here's what I've presently got, and some related notes:
// as of 2024-09-23:
// TTL algorithm zsk bits ksk bits
// default parent DS 86400 13
// 4.0.1.0.0.2.ip6.arpa DNSKEY 86400 8 2048 1024
// com parent DS 86400 13
// in parent DS 3600 8 2048 1024
// net parent DS 86400 13
// org parent DS 3600 8 2048 1024
dnssec-policy "my_8_2048_1024" { # legacy
keys {
ksk key-directory lifetime unlimited algorithm 8 2048;
zsk key-directory lifetime 30d algorithm 8 1024;
};
nsec3param;
};
dnssec-policy "my_13_8_2048_1024" { # transition
keys {
ksk key-directory lifetime unlimited algorithm 13;
zsk key-directory lifetime 30d algorithm 13;
ksk key-directory lifetime unlimited algorithm 8 2048;
zsk key-directory lifetime 30d algorithm 8 1024;
};
nsec3param;
};
dnssec-policy "my_8_2048_1024_ds_ttl_3600" { # in org
keys {
ksk key-directory lifetime unlimited algorithm 8 2048;
zsk key-directory lifetime 30d algorithm 8 1024;
};
parent-ds-ttl 3600;
nsec3param;
};
dnssec-policy "my_default" {
keys {
ksk key-directory lifetime unlimited algorithm 13;
zsk key-directory lifetime 30d algorithm 13;
};
nsec3param;
};
And thus far, on that primary, these are the domains that have
dnssec-policy and the policy they're set to:
my_8_2048_1024 e.9.1.0.5.0.f.1.0.7.4.0.1.0.0.2.ip6.arpa
my_13_8_2048_1024 sf-lug.com
my_13_8_2048_1024 sflug.com
my_8_2048_1024_ds_ttl_3600 savingthedolph.in
my_13_8_2048_1024 sf-lug.net
my_13_8_2048_1024 sflug.net
my_8_2048_1024_ds_ttl_3600 digitalwitness.org
my_8_2048_1024_ds_ttl_3600 sflug.org
$
On Mon, Sep 23, 2024 at 10:53 PM Al <aw009(a)sunnyside.com> wrote:
>
> On the way to bed, and no time to really dig into this now. I'll have
> to have a better look tomorrow.
> Seems like something is amiss on my ns0 (aka nsx) but not on ns1 or ns2
> (aka nsy). (all .sunnyside.com)
>
> ns0/nsx sample logs:
> root@post:/var/log/named# tail -f *.log | grep -i sflug
> 23-Sep-2024 22:39:25.360 lame-servers: info: lame server resolving
> 'sflug.org' (in 'sflug.org'?): 2603:3024:180d:f100:50:242:105:34#53
> 23-Sep-2024 22:39:25.368 lame-servers: info: insecurity proof failed
> resolving 'sflug.org/NS/IN': 2001:470:1f04:51a::2#53
> 23-Sep-2024 22:39:25.392 lame-servers: info: lame server resolving
> 'sflug.org' (in 'sflug.org'?): 2600:1f1c:528:c500:5e0b:8a37:6598:356c#53
> 23-Sep-2024 22:39:25.416 lame-servers: info: no valid RRSIG resolving
> 'sflug.org/NS/IN': 2001:470:1f05:19e::3#53
> 23-Sep-2024 22:39:25.440 lame-servers: info: no valid RRSIG resolving
> 'sflug.org/NS/IN': 96.95.217.99#53
> 23-Sep-2024 22:39:25.440 lame-servers: info: lame server resolving
> 'sflug.org' (in 'sflug.org'?): 50.242.105.52#53
> 23-Sep-2024 22:39:25.456 lame-servers: info: no valid RRSIG resolving
> 'sflug.org/NS/IN': 198.144.194.12#53
> 23-Sep-2024 22:39:25.476 lame-servers: info: lame server resolving
> 'sflug.org' (in 'sflug.org'?): 50.18.139.240#53
> 23-Sep-2024 22:39:25.504 lame-servers: info: no valid RRSIG resolving
> 'sflug.org/NS/IN': 96.86.170.229#53
> 23-Sep-2024 22:39:25.368 dnssec: info: view internal: validating
> sflug.org/NS: no valid signature found
> 23-Sep-2024 22:39:25.416 dnssec: info: view internal: validating
> sflug.org/NS: no valid signature found
> 23-Sep-2024 22:39:25.440 dnssec: info: view internal: validating
> sflug.org/NS: no valid signature found
> 23-Sep-2024 22:39:25.456 dnssec: info: view internal: validating
> sflug.org/NS: no valid signature found
> 23-Sep-2024 22:39:25.504 dnssec: info: view internal: validating
> sflug.org/NS: no valid signature found
>
> All three machines show:
> sf-lug.org. 3591 IN SOA ns0.sf-lug.org.
> Michael\.Paoli.berkeley.edu. 1727012375 10800 3600 3600000 86400
>
> so at least they should be in sync.
>
> I got this failure:
> al@post:/z/dns$ dig ns sflug.org
>
> ; <<>> DiG 9.16.6 <<>> ns sflug.org
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 27626
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 1472
> ; COOKIE: 6b64e1cd9d59aeff0100000066f2508d5ba5e929bf62035f (good)
> ;; QUESTION SECTION:
> ;sflug.org. IN NS
>
> ;; Query time: 155 msec
> ;; SERVER: 192.147.248.10#53(192.147.248.10)
> ;; WHEN: Mon Sep 23 22:39:25 PDT 2024
> ;; MSG SIZE rcvd: 66
>
>
> But a few minutes later it was fine. So whatever it is, it's intermittent.
>
> More later...
> Al
So,
I'd much earlier noted on my calendar:
2024-09-24 >= 10:56AM US/Pacific PST8PDT -07:00, if Comcast Business
hasn't screwed up DNS with linuxmafia.com for more than a year,
probably time to remove the
port 5353 work-arounds.
But, I don't think (year earlier) was
necessarily the last of those Comcast DNS issues.
So, last I see that hints of that
is chatter/discussions that tapered off
around 2024-06-04
https://lists.balug.org/mailman3/hyperkitty/list/balug-admin@lists.balug.or…
So, as earlier, since wasn't our first go-round,
I'll push the end date of that work-around to
>~=2025-06-05
and if it's by then had no more such glitches,
will then drop that work-around.
In the meantime, doesn't really hurt hardly anything,
and easier to already have that workaround in place,
should it be needed at all before then.
So, calendar updated:
2025-06-25 if Comcast Business hasn't screwed up DNS with
linuxmafia.com for more than a year, probably time to remove the port
5353 work-arounds.
After unclean shutdown of mailman3,
even when (re)started with
start --force
mailman3 may fail to remove the lock files and thus fail to (re)start.
Details on (I submitted as):
https://gitlab.com/mailman/mailman/-/issues/1174
I'm definitely not the first person to have noticed this issue.
Hopefully they'll get around to fixing it (it's been over 9 years ...
but they may have lacked sufficient relevant detail in the
earlier reports).
Manual workaround:
remove the lock files in
/var/lib/mailman3/locks/
after checking to ensure that
none of them reference and/or are used by any PID(s) still relevant to mailman3
One thing I do like about Mailman 3 is all the available message
acceptance settings, both for list default, and per user.
Those include:
List default -- follow the list's default member action.
Hold -- This holds the message for approval by the list moderators.
Reject -- this automatically rejects the message by sending a bounce
notice to the post's author. The text of the bounce notice can be
configured by you.
Discard -- this simply discards the message, with no notice sent to the
post's author.
Accept -- accepts any postings without any further checks.
Default Processing -- run additional checks and accept the message.
Most notably - Discard highly useful to set non-members to Discard.
Likewise for moderated list(s) where few are allowed to post (e.g.
balug-announce(a)lists.balug.org).
Makes listadmin life fair bit easier, as most all such posting attempts,
that would otherwise be held for moderation on Mailman 2, can be simply
discarded, as almost all of them are junk/garbage/spam and the like -
maybe only about once a year or so does a legitimate message come in
that way. And Reject really isn't that useful, as that happens after
the email has already been received, rather than at SMTP time, so there
would be backscatter issues with that (the overwhelming number of such
mail is spam, and those are typically rather to quite forged senders,
victim compromised accounts, etc.).
And ... not sure exactly how much I (dis)like it on Mailman 3,
but Mailman 2 data was mostly in python 2 pickle (.pck) format files,
with data/configuration bits also being in flat (and also html) files.
Well, Mailman 3, while some configuration bits are stored in flat
configuration files, for better and/or worse, most of the data is stored
in an actual relatively proper database (e.g. sqlite).
So, balug-announce(a)lists.balug.org e.g. imported from Mailman 2,
most all users had the moderation set to hold (Mailman 2 didn't have a
discard option). On Mailman 3 I set the list default to discard for
that list. But almost all users were imported from Mailman 2 and that
import preserved their hold setting. I wanted to change it for those
users to the list default (discard). A bit to chase it down in the
database, but once found and reasonably tested, one SQL statement to do
the needed:
UPDATE member SET moderation_action = NULL WHERE list_id =
'balug-announce.lists.balug.org' and moderation_action = 0;
... and with that done for 492 users. Yay!(?)
Not super easy/pleasant but ... well, compared to Mailman 2,
yeah I think it's at least fair bit better than Mailman 2,
at least on that point/feature/capability.
FYI, did send this out:
---------- Forwarded message ---------
From: Michael Paoli <michael.paoli(a)berkeley.edu>
Date: Mon, Jul 15, 2024 at 11:54 PM
Subject: Mailman 3 listadmin testing (and potential hosting) available
To: Grant Bowman <grantbow(a)gmail.com>, Grant Bowman
<grantbow(a)dvlug.org>, Aaron <acohen36(a)gmail.com>, Tom
<tomrlopes(a)gmail.com>, Ian <droid1836(a)gmail.com>, Rick Moen
<rick(a)linuxmafia.com>, Al <aw009(a)sunnyside.com>
Various SF Bay Area [L]UG(s) [potential] list admin folk(s),
May potentially be some additional folks (if interested, contact me).
Thought I'd invite, for possible testing, etc. some folks that are
list admins of various SF Bay Area [L]UG(s), or might be potentially
very seriously interested in that.
Most notably, have Mailman 3 fairly well set up (at least metastable)
on the BALUG VM (still have more Mailman 2 --> Mailman 3 migration
steps to fully test out and validate before I do all those migrations
for the production lists, including related infrastructure of automation
and information, etc.).
In those regards, if folks are interested in a test list on there for
potentially testing for [L]UG use and list administration thereof,
just let me know. Note that there are test list(s) there currently
(notably balug-test3(a)lists.balug.org which is on Mailman 3 and will
eventually become and be merged into balug-test(a)lists.balug.org
when that gets migrated to Mailman 3), but I was thinking more
significantly and distinct from that, list(s) where I could grant folks
full listadmin access and control. E.g.:
sf-lug-test(a)lists.balug.org
dvlug-test(a)lists.balug.org
berkeleylug-test(a)lists.balug.org
etc.
Could even do different (sub)domains, e.g.:
@lists.dvlug.org
@lists.sf-lug.org
etc., though subdomains would require some fair bit of additional
work/coordination to set that all up properly (DNS, SPF, TLS, ...)
so @lists.balug.org would be the simpler easier place to start and is
already available.
Anyway, if interested, let me know (and notably list name).
If you merely want to test list, can use the existing
balug-test3(a)lists.balug.org e.g.
send email to: balug-test3-subscribe(a)lists.balug.org to subscribe
or see:
https://lists.balug.org/mailman3/postorius/lists/balug-test3.lists.balug.or…
And yes, for at least many (SF [L]UG lists), I'd be willing to host on
the BALUG VM (and in general they could have their own subdomain,
in fact for actual general production use that would be exceedingly
recommended, e.g. something like, say for SF-LUG:
sf-lug(a)lists.sf-lug.org
sf-lug-test(a)lists.sf-lug.org)
But even if one mostly wants to reasonably well test out Mailman 3 and
list administration thereof, the BALUG VM may be a useful place to be
able to reasonably well do that (want to test drive Mailman 3 and see if
it might make suitable list infrastructure for your [L]UG?).
> From: Michael Paoli <michael.paoli(a)berkeley.edu>
> Date: Sat, Jun 29, 2024 at 1:52 PM
> Subject: Re: dvlug list
> To: Grant Bowman <grantbow(a)gmail.com>
> Cc: Diablo Valley Linux Users Group (DVLUG) <dvlug(a)linuxmafia.com>,
> Diablo Valley Linux Users Group (DVLUG) <list(a)dvlug.org>
>
> Grant,
...
> Oh, ... I'll also hang it out there for your consideration, if you
> may be interested ... if you may possibly want to have subdomain with
> DVLUG list(s) on <whatever>@lists.dvlug.org., I could potentially
> host that if you so wanted that ... but maybe ask me again in a
> couple weeks or so ... I'm still doing migration off of mailman
> version 2 (could do mailman 2 in the meantime if you really wanted
> that), so still have more testing, etc. to do, and want to wait a bit
> 'till that's all done and the dust well settled before I consider
> that migration to be a full success (in case I find any latent issues
> that may not be immediately apparent). I'd also think Rick might
> even be willing to do same (notably subdomain part for list) on the
> existing linuxmafia.com host. In any case, food for thought. Could
> also potentially migrate so all the old (most notably archives) would
> still also be present under lists.dvlug.org (similar to how things
> are for BALUG under lists.balug.org). Anyway, just a thought, and
> thought I'd hang the offer out there if you may be interested (and
> would also have the capability to access and back up full archives,
> also member subscription list). And, yeah just to think about it
> more generally, if you use a subdomain (e.g. lists.dvlug.org) for
> list(s), rather than the main domain (dvlug.org), that does leave
> things much more flexible for how one may want to handle list(s),
> most notably it can be hosted or the like, exceedingly independent of
> whatever else may be going on for hosting or whatever on
> [www.]dvlug.org (e.g. web site, etc.).