/var space issues on balug VM, almost certainly from Internet
miscreants & their bots
So ... bit more background on this below,
and see also wiki page:
https://www.wiki.balug.org/wiki/doku.php?id=system:annoyances
I'll likely have the more detailed analysis/summary/progress
there - including also history of the wiki page.
I'll likely post more to
BALUG-Admin <balug-admin(a)lists.balug.org>
when it's more-or-less resloved ... or other major or possibly
significant developments/findings/changes.
Anyway, feel free to ask/discuss on BALUG-Admin if you may wish (or toss
about ideas/suggestions/questions/whatever).
Anyway, I thought the list traffic would be more fitting for
BALUG-Admin than BALUG-Talk, so ... pointing that discussion
to here ... also linked (notably on wiki page) to the earlier,
if one wants to catch up on reading that (much or possibly all of which,
might also be captured below).
references/excerpts:
> From: "Michael Paoli" <Michael.Paoli(a)cal.berkeley.edu>
> Subject: -->BALUG-Admin Re: [BALUG-Talk] Possible solution of
> blankness or editing outline-only on BALUG DokuWiki?
> Date: Fri, 23 Apr 2021 21:40:57 -0700
> I'm going to suggest taking this discussion to the
> BALUG-Admin <BALUG-Admin(a)lists.balug.org>
> https://lists.balug.org/cgi-bin/mailman/listinfo/balug-admin
> list, as,
> though the matter does involved Linux somewhat generally, it's much
> more specific and relevant to BALUG and the specific inner
> workings thereof. So, expect my next update and further communications
> on the matter to be on the BALUG-Admin list, and I'll likely start
> it off with
> latest update(s) and some background references/links on the matter, at least
> as of late.
>
> Also ...
> dates, lists, number of members:
> YYYY-MM-DD announce talk admin
> 2021-04-01 832 259 35
> https://lists.balug.org/pipermail/balug-admin/2021-April/001046.html
> I'm guestmating probably not around 259 subscribers are that interested
> in these particularly details ... but quite possibly closer to about
> 35 are ... or would be.
> I may well also put up a wiki page regarding findings/developments ...
> or maybe update an older one if such suitable wiki page already exists.
>
> Anyway, I still have some 'work' to do ...
> # date -Iseconds && df -h /var
> 2021-04-24T04:39:59+00:00
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/balug-var 6.4G 5.9G 287M 96% /var
> #
>
>> From: "Michael Paoli" <Michael.Paoli(a)cal.berkeley.edu>
>> Subject: Re: [BALUG-Talk] Possible solution of blankness or editing
>> outline-only on BALUG DokuWiki?
>> Date: Fri, 23 Apr 2021 08:44:22 -0700
>
>> Well, not out of the woods yet.
>>
>> Have done fair bit of compression (even using xz -9) of some
>> moderately older data on /var ... and have grown /var several
>> times, ... yet it filled up again very recently (or exceedingly
>> close - I saw it at 100% again, but that's probably rounded, so
>> at least >99.5%).
>>
>> So, at present, now have:
>> # df -h /var
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/mapper/balug-var 6.4G 5.7G 491M 93% /var
>> #
>> And very recently (only few days back) that was:
>>> df -h /var
>>> Filesystem Size Used Avail Use% Mounted on
>>> /dev/mapper/balug-var 4.9G 4.5G 190M 97% /var
>> Can't just keep growing /var indefinitely.
>> Much of the volume is mail and related - logs, queued mail,
>> bounces & failures, mail to listadmin and postmaster, etc., and mostly
>> lots of other nuisance stuff - fail2ban database takes fair bit of space,
>> but trimming what that holds would increase data in other logs - notably
>> of lots of nuisance failed login/authentication attempts and the like.
>>
>> And, even if I caught it before completely full, some services (e.g.
>> exim4) will proactively stop processing when the free filesystem
>> space runs too low.
>>
>> So, yeah, I'll need to do some more work on it. May need to analyze why
>> it's ballooning more than it was in past - what's the nature of the
>> problematic traffic that keeps driving up the storage.
>>
>> I suspect it's mostly spam (and attempts thereof) and bad bots
>> doing/attempting stuff ... essentially all some type(s) of nefarious
>> unwanted traffic chewing up resources.
>>
>> So, in only a couple days or so, it's already nearly 1.5 GiB more data,
>> and that's already after having done fair bit of aggressive data
>> data compression on some older data (e.g. some of the older mail data
>> easily compressed from around 200 of MiB to under a few tens of MB ...
>> and I even moved the compressed off of the /var filesystem).
>>
>> So, ..., yeah, miscreants on 'da Internet ... more work for me to do. :-/
>>
>> Would also be nice to get some of that /var space back, and not have to give
>> so much of it over to what's basically mostly unintersting crud.
>>
>> And, have had stuff like this happen before, ... e.g. like bad bots trying
>> to register on wiki - and chewing up huge volumes of space ... that's why
>> there's no self-registration option anymore ... blame it on abusers on
>> 'da Internet. Manual registration only now. Even captcha's weren't really
>> enough for most of that ... bad bots still pounded away and repeatedly
>> filled up log space. Maybe more fail2ban config would help that
>> ... but then
>> that also ends up chewing up additional space with its database ...
>>
>> # du -x /var | sort -bnr | head -n 12
>> 5866812 /var
>> 2102592 /var/lib
>> 1249060 /var/log
>> 1017624 /var/spool
>> 1015100 /var/mail
>> 932200 /var/spool/exim4
>> 731780 /var/lib/fail2ban
>> 616584 /var/lib/clamav
>> 531456 /var/log/apache2
>> 505952 /var/log/exim4
>> 491980 /var/spool/exim4/input
>> 364892 /var/spool/exim4/msglog
>> #
>>
>>> From: "aaronco36 via BALUG-Talk" <balug-talk(a)lists.balug.org>
>>> Subject: Re: [BALUG-Talk] Possible solution of blankness or
>>> editing outline-only on BALUG DokuWiki?
>>> Date: Thu, 22 Apr 2021 16:21:14 +0000 (UTC)
>>
>>> Quoting Michael Paoli <Michael.Paoli at cal.berkeley.edu> from [1]:
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> Thanks for the report, but I'm unable to reproduce the issue.
>>> Anyone else able to reproduce it? Are you/anyone seeing the
>>> issue at this time? Anyone else also see it earlier?
>>>
>>> I've tried the following on:
>>> https://www.wiki.balug.org/wiki/doku.php without being able to
>>> reproduce the issue:
>>> Chromium
>>> Chromium incognito mode
>>> Firefox
>>> Firefox private browsing mode
>>> Lynx
>>> curl
>>> wget
>>> Chome on Android from another ISP
>>> Chrom icognito mode on Android from another ISP
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> Cut-and-pasted a text screen-capture and submitted it as a
>>> [BALUG-Test] 'Test of BALUG Wiki pages' posting yesterday morning;
>>> see [2].
>>> AFAICT, that "blankness or editing outline-only issue" for BALUG
>>> DokuWiki pages [3] and [4] occurred from before 08:00am PDT up
>>> through the late morning, and then went away by mid-afternoon
>>> yesterday.
>>> Probably reloaded both those BALUG DokuWiki pages a half-dozen
>>> times or so over the morning time-period using mostly Firefox-ESR
>>> standard mode and Firefox-ESR private browsing mode, and maybe
>>> once or twice using TOR browser in its [least] Safe mode.
>>>
>>> Same results for all browsers used to test those pair of pages
>>> during that entire morning time-period.
>>>
>>>
>>> Quoting Michael Paoli <Michael.Paoli at cal.berkeley.edu> from [5]:
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> That may have done it.
>>> Looks like I need to add more space to /var and/or clean some more stuff
>>> off of it:
>>> df -h /var
>>> Filesystem Size Used Avail Use% Mounted on
>>> /dev/mapper/balug-var 4.9G 4.5G 190M 97% /var
>>> #
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> Thanks much, Michael, for effectively diagnosing and resolving the
>>> issue! :-)
>>>
>>> Besides considering the issue arising from the DokuWiki site
>>> itself or BALUG's device /dev/mapper/balug-var, was also
>>> considering the possibility that the issue might have arisen from
>>> my own browser client.
>>>
>>> Why so?
>>>
>>> Well, there was no issue whatsoever with successfully loading
>>> BALUG DokuWiki pages [3] and [4] throughout Sunday and Monday of
>>> this week.
>>> What changed on Tuesday afternoon was upgrading Firefox-ESR to a
>>> slightly higher, minor version as per 'less
>>> /var/log/apt/history.log' :
>>>> Start-Date: 2021-04-20 14:10:03
>>>> Commandline: apt upgrade
>>>> Upgrade: firefox-esr:amd64 (78.9.0esr-1~deb10u1, 78.10.0esr-1~deb10u1)
>>>> End-Date: 2021-04-20 14:10:15
>>>
>>> One or more bugs in the upgraded FF 78.10.0esr could have
>>> conceivably rendered text and links in the BALUG DokuWiki
>>> incorrectly afterwards, as remote as this possibility might seem
>>> to be.
>>> It's good to know that this wasn't the case at all, and that this
>>> latest Firefox-ESR browser was fine after all :-)
>>>
>>> Thanks again Michael!
>>>
>>> -Aaron
>>>
>>> ===========================
>>> REFERENCES
>>> ===========================
>>> [1]https://lists.balug.org/pipermail/balug-talk/2021-April/000220.html
>>> [2]https://lists.balug.org/pipermail/balug-test/2021/000048.html
>>> [3]https://www.wiki.balug.org/wiki/doku.php
>>> [4]https://www.wiki.balug.org/wiki/doku.php?id=balug:offered_wanted_hardware…
>>> [5]https://lists.balug.org/pipermail/balug-talk/2021-April/000221.html
>>> ===========================
Some web stats by domains, etc.
I was interested, thought others might be curious/interested too, so ...
I've got the log rotation set up so it retains bit over a year's worth of
web server logs
$ awk '{if($1=="rotate" || $1 !~ /^#/ && $1 ~ /ly$/)print}'
/etc/logrotate.d/apache2
weekly
rotate 60
so, fair bit of data available.
This web server runs on the balug Virtual Machine (VM) host,
which covers for not only the BALUG [Linux] User Group ([L]UG),
but others too. Anyway, grabbed the data from the logs, and did a wee
bit of analysis. This is from the {ssl_,}access.log* files.
So, basically analyzed domain and port, here's the information/analysis,
and counts, from the highest levels on down to that:
13945791 Total
That's the total traffic seen, all sites/domains that got logged.
That's approximately:
232430 per week (13945791/60)
33204 per day (13945791/60/7)
1384 per hour (13945791/60/7/24)
23 per minute (13945791/60/7/24/60)
0.38 per second (13945791/60/7/24/60/60)
So about one per 2.6 seconds (1/(13945791/60/7/24/60/60))
by [L]UG or project or whatever and the like. Note also that test
traffic isn't excluded, so, e.g. some domains that aren't in DNS or
so delegated (or not yet or no longer in DNS) also show wee bit 'o
traffic.
9504843 BALUG
3463857 BerkeleyLUG
968724 SF-LUG
8283 digitalwitness.org
51 BAD
33 BUUG
In a bit more detail, by relevant TLD:
9504843 balug.org
3457611 berkeleylug.com
896560 sf-lug.org
40847 sf-lug.com
13698 sflug.org
8283 digitalwitness.org
7458 sflug.net
6246 berkeleylug.org
6084 sflug.com
4077 sf-lug.net
51 bad.debian.net
33 buug.org
Note that for all of SF-LUG's various domains,
traffic drops by >20x once we leave the canonical TLD ([www.]sf-lug.org)
and drop to 1st runner up in SF-LUG traffic, >65x to 2nd runner up, and
> 120x for 3rd runner up.
For BerkeleyLUG, the obscure and almost completely unknown (was never
canonical nor particularly promoted) BerkeleyLUG.org,
The canonical BerkeleyLUG.com has >550x the traffic ...
however BerkeleyLUG.org no longer exists, so it was something >1/550 of
the traffic when it still existed (but even back then I recall it being
relatively negligible portion of traffic).
So, by domain ... secure.balug.org is somewhat surprising. It's probably
mostly from "bad" or not so well behaved bots, poking at the BALUG wiki,
trying to login there, and getting redirected ... or maybe such bots
thinking there's something more "interesting" to go after because it's
got "secure" in the name? Maybe ought phase that one out, not nearly
so relevant anymore. I think the original idea is that one would
redirect to force https, but I believe now all the domains offer https,
and as/where relevant (e.g. wiki login and after doing so with cookie
that has authenticated state) redirection can be done from http to
https with same domain - no need for a separate domain for that.
Also a bit surprising too, is BerkeleyLUG.com being as high as it is.
It's WordPress, and bots - legitimate, or not so, and search engines,
well, WordPress effectively "expands" to a quite huge set of unique
URLs, even if the content of each isn't all that incredibly unique.
E.g. I remember earlier trying to crawl the site when it was still
hosted by WordPress.com - it expanded into a huge amount of content
that wasn't particularly feasible to replicate in-place as-is without
WordPress - especially relatively to the actual full data in/behind
the site - a much more manageable and smaller set of data. So that
may, at least partially, explain the somewhat surprising large number
there.
Likewise, BALUG's wiki - lots of content - especially if one crawls the
archive of all older versions of all pages.
And BALUG's list - every posting can be individually crawled, so, lots of
URLs, and probably keeps search engines / bots relatively busy.
And one other that's slightly surprising ... the mx entries.
Really noting promoting - or even configured for that as web,
other than it existing in DNS. So that's probably mostly "bad" bots
and/or a bit of test traffic.
Also, pi.berkeleylug.com is slightly, but not entirely redundant with
https://berkeleylug.com/Pi.BerkeleyLUG/
Notably pi.berkeleylug.com 302 redirects to the above ...
however also, there's account and sudo and dynamic DNS highly available
to pi.berkelelylug - so if/whenever they / that SIG wants it to go to
somewhere else in DNS, it's readily available for that. But if the
DNS & web traffic hits the BALUG host's web server, it 302 ("temporary")
redirects as noted above.
4467098 secure.balug.org
3408433 berkeleylug.com
2083824 www.balug.org
1692238 www.wiki.balug.org
998636 lists.balug.org
875806 www.sf-lug.org
185888 balug.org
47829 www.berkeleylug.com
42223 www.archive.balug.org
23561 www.sf-lug.com
17286 sf-lug.com
15402 sf-lug.org
14875 old-debian.balug.org
8993 sflug.org
5372 www.digitalwitness.org
5175 wiki.balug.org
5047 berkeleylug.org
4705 www.sflug.org
4295 sflug.com
4052 sflug.net
3841 www.ipv4.sf-lug.org
3406 www.sflug.net
3291 www.new.balug.org
3291 sf-lug.net
2911 digitalwitness.org
2819 www.test.balug.org
2519 www.ipv4.balug.org
2016 www.beta.balug.org
1789 www.sflug.com
1225 www.php.test.balug.org
1199 www.berkeleylug.org
1182 ipv4.sf-lug.org
1096 mx.balug.org
966 ipv4.balug.org
786 www.sf-lug.net
753 www.pi.berkeleylug.com
611 mx.lists.balug.org
596 pi.berkeleylug.com
266 www.ipv6.balug.org
262 www.ipv6.sf-lug.org
77 ipv6.balug.org
67 ipv6.sf-lug.org
51 bad.debian.net
25 www.buug.org
8 buug.org
And, nothing horribly surprising here, given the above. This is
essentially same again, but broken down by by port.
Also, if we total up all of http (:80) and https (:443) we get:
7263155 :80 (http)
6682636 :443 (https)
And by domain and port:
3570578 secure.balug.org:80
2942325 berkeleylug.com:443
1762326 www.balug.org:443
1355057 www.wiki.balug.org:80
896520 secure.balug.org:443
834027 www.sf-lug.org:80
616148 lists.balug.org:443
466108 berkeleylug.com:80
382488 lists.balug.org:80
337181 www.wiki.balug.org:443
321498 www.balug.org:80
140573 balug.org:80
45315 balug.org:443
43816 www.berkeleylug.com:80
41779 www.sf-lug.org:443
29510 www.archive.balug.org:80
22723 www.sf-lug.com:80
15730 sf-lug.com:80
14827 old-debian.balug.org:80
12713 www.archive.balug.org:443
11427 sf-lug.org:80
6618 sflug.org:80
4604 berkeleylug.org:80
4420 www.digitalwitness.org:80
4134 wiki.balug.org:80
4013 www.berkeleylug.com:443
3975 sf-lug.org:443
3548 sflug.com:80
3263 sflug.net:80
3251 www.sflug.org:80
2738 sf-lug.net:80
2485 digitalwitness.org:80
2462 www.test.balug.org:80
2394 www.ipv4.sf-lug.org:80
2375 sflug.org:443
2244 www.new.balug.org:80
1964 www.sflug.net:80
1767 www.beta.balug.org:80
1556 sf-lug.com:443
1454 www.sflug.org:443
1447 www.ipv4.sf-lug.org:443
1442 www.sflug.net:443
1409 www.sflug.com:80
1392 www.ipv4.balug.org:443
1127 www.ipv4.balug.org:80
1047 www.new.balug.org:443
1041 wiki.balug.org:443
975 www.php.test.balug.org:80
952 www.digitalwitness.org:443
879 ipv4.sf-lug.org:80
838 www.sf-lug.com:443
789 sflug.net:443
787 www.berkeleylug.org:80
747 sflug.com:443
736 ipv4.balug.org:80
718 www.pi.berkeleylug.com:80
599 mx.balug.org:80
553 sf-lug.net:443
497 mx.balug.org:443
463 www.sf-lug.net:80
443 berkeleylug.org:443
426 digitalwitness.org:443
412 www.berkeleylug.org:443
381 mx.lists.balug.org:80
380 www.sflug.com:443
376 pi.berkeleylug.com:80
357 www.test.balug.org:443
323 www.sf-lug.net:443
303 ipv4.sf-lug.org:443
250 www.php.test.balug.org:443
249 www.beta.balug.org:443
230 mx.lists.balug.org:443
230 ipv4.balug.org:443
220 pi.berkeleylug.com:443
194 www.ipv6.sf-lug.org:80
173 www.ipv6.balug.org:80
93 www.ipv6.balug.org:443
68 www.ipv6.sf-lug.org:443
67 ipv6.balug.org:443
60 ipv6.sf-lug.org:443
48 old-debian.balug.org:443
45 bad.debian.net:80
35 www.pi.berkeleylug.com:443
18 www.buug.org:80
10 ipv6.balug.org:80
7 www.buug.org:443
7 ipv6.sf-lug.org:80
6 bad.debian.net:443
4 buug.org:80
4 buug.org:443
Anyway, maybe some day I'll get some "real" web reporting in place.
It is on the todo list, but ...
$ wc todo
6156 22307 188878 todo
$
It's also not a FIFO list, nor FILO.
It's mostly a priority interrupt driven stack that gets a lot
of reordering applied to it, and tends to grow much more than it
shrinks. Well, at least I don't have to worry about running
out of stuff to do - have well over lifetime's worth 'o stuff on the
list.
More BALUG and [L]UG specific lists may be found on BALUG's wiki,
though they're not necessarily complete nor current.