debian 93

Did not do anything seriously nasty that i was immediately aware of and the systemd fault mentioned here (my blog) means systemd cannot load postfix on start up at all systend is very shit software.

zeitgeist-daemon is another headache with systemd as i baked a disk and could not start the x server on the machine on next login.  So i killed all my users processes [not root] and gave up and found some hardware elsewhere that worked.  I shall look into removing zeitgeist-daemon

I think i also lost x access for root in 9.2 as well not that i use it often [once a year] but having a gui as root is nicer than vi when you have four servers to do stuff to..

I could see a migration to the bsd way if things continue to go a certain way.- I demand very little but when systemd wont start stuff,magic daemons stop x and none of it your fault then perhaps linux as a server is not the thing i thought it was.

On that thought lets leave as accommodating linux quirks is possible but not that desirable perhaps you can justify calling /usr/sbin/postfix instead of using systemd in scripts but i feel perhaps there is something that works rather than is supposed too but does not.

I look forward to exploring bsd and knowing more so i can make an informed choice.

mailgraph and logwatch reporting curiosities with postmulti and some regex’es for fun.

The zoo’s mailgraph charts are not working and i have mentioned it before (my blog).

So after changing our /etc/postfix instance (we have more better instances) for a new feature to allow outbound internet mail to be sent to an address the charts began to show only that traffic.  Bounces too also appear to work (not shown).

Spam and viruses as defined by amavis do work but the received email from those other postfix instances is still not being recognised even with explicit syslog statements in the main.cf file.

So something is off

Reading the charts could give you the impression that despite receiving email that the chart does not graph bur we appear send out spam and viruses and blocked,  The bounces where something i induced and could have been dmarc related too as many dmarc reporters have problems clearing there gmail inbox..

It is a good reminder that badly made statistics may look interesting but do not reflect reality.

The logwatch config files /usr/share/logwatch/default.conf/services/postfix.conf are written as perl and at this point are beyond my comprehension

*OnlyService = “(?:post(?:fix|grey|fwd|fix-1|fix2|fix-0|fix-3|policyd-spf)(?:/[-\w]*)?”
$postfix_Syslog_Name = “(?:post(?:fix|grey|fwd)|policyd-spf)”
# POSTMULTI NOT WORK *OnlyService = “postfix\d?/[-a-zA-Z\d]*”
#$postfix_Syslog_Name = “postfix\d?”

My changes are in bold. That does not work.. /etc/postfix-1 etc is how postmulti expects its managed instances to be located (my blog).

A few days pass and with the help of a pcre debuger [https://regex101.com/] i find that

$postfix_Syslog_Name = “postfix/[\w]*”
*OnlyService = “(:postfix-1/|postfix-2/|postfix-3/|postfix-4/|policyd-spf|postfix/|post-grey|post-fwd)(?:[-\w]*)?”

Provides output from postmulti instances as well as the /etc/postfix daemon.  I might not need that last postfix on the third line but completist me me thought it worth specifying.

post-fwd and post-grey are not used here in the zoo we use postscreen  The spf log part of the the section is a little unwieldy but that always was and i could turn it off,

I find with postmulti reporting that “postfix/lmtp” is best stated as “lmtp” if grepping unless you want to add extra grep lines to your cron jobs.

So charts are still a bit messed up.   Not the end of the world although i have cron jobs that grep for connections and sasl abusers so between the broken things and our existing zoo cron jobs we keep on top on what postfix is having to deal with.

A work in progress mailgraph.requires that the /usr/sbin/mailgraph file be changed for postmulti.

I seemed ho have some luck and you can see the switch on since the data before was sent from a non internet postfix host denoted by green and red suddenly appearing.

I changed the line for postfix (a regex again) from

if($prog =~ /^postfix\/(.*)/) {

to

if($prog =~ /:postfix|postfix-1\/(.*)|postfix-2\/(.*)|postfix-3\/(.*)|postfix-4\/(.*)/) {

Which is not very maintainable and a bit of a bodge job but gets the regex working for more than one instance..  If that reflects reality or not i will have to check with logwatch reporting although with postfix dropping more bad connections earlier (my blog) feels right so the charts now ignore a large quantity of data of bad smtp clients say.

106 Reject by IP --------
 3 49.213.57.100 unknown
 3 103.241.75.75 unknown

So mailgraph and postfix seem now not count certain items compared to before the upgrade.  So that regex might see an edit.

Mailgraph was and then was not working i was unsure of my efforts – another regex to adjust

I eventually found

/postfix-1\/(.*)|postfix-2\/(.*)|postfix-3\/(.*)|postfix-4\/(.*)|postfix/

Appears to show green / blue and red posfix lines

Fail2ban also seems to need some help – although it seems it will not trip with rate throttling controls in my experience although the odd prober does try an extract from logwatch.

smtp
10 AUTH command rate
10 110.175.112.118 110-175-112-118.tpgi.com.au
1 Connection rate
1 110.175.112.118 110-175-112-118.tpgi.com.au

Perhaps fail2ban’s postfix jails are redundant with the rate limiting feature in newer postfix. Not that fail2ban tripped that often with our non postmulti config.

As most of our email traffic is using tls (dane – my blog) (or trying to) i somehow think mailgraphs use out of the box does not reflect reality with the rate controls, bad clients getting ignore and tls traffic not shown so i suppose this graph shows genuine email traffic rather than all port 25 attempts..

Further Debian Stretch as a server notes

rounding up the fairies

Following on from this (my blog) i continue my bug upgrade hunt.  Its not over.

I have mentioned many of these items before in this blog, it is not my job to tell you what they are.

Apache/Perl

Rkhunter say:

Warning: The command ‘/usr/bin/lwp-request’ has been replaced by a script: /usr/bin/lwp-request:
Perl script text executable

Might explain why perl did not exec via my ‘old’ cgi scripts as Jessie

Opendkim /Postfix

I ‘needed’ an extra line (also in /etc/default/opendkim)

PidFile /var/run/opendkim/opendkim.pid

in opendkim.conf – mail was being sent without dkim

I appear to not have dkim signatures in outbound email., opendkim-testkey thinks its config is good  i think it might be easier to reconfigure postfix from scratch.  It is not milter_protocol= 6 and 2 does not work.  Um no idea.   Opendkim seems up but not connected.

Opendkim was not working. Eventually this clued (not here) me in that the openkim config files where fine but the systemd script was buggered

So if your config files are right but the daemon refuses to follow orders try this

edit /lib/systemd/system/opendkim.service

from this

[Unit]
Description=OpenDKIM DomainKeys Identified Mail (DKIM) Milter
Documentation=man:opendkim(8) man:opendkim.conf(5) man:opendkim-genkey(8) man:opendkim-genzone(8) man:opendkim-testadsp(8) man:opendkim-testkey http://www.opendkim.org/docs.html
After=network.target nss-lookup.target

[Service]
Type=forking
PIDFile=/var/run/opendkim/opendkim.pid
User=opendkim
UMask=0007
ExecStart=/usr/sbin/opendkim -P /var/run/opendkim/opendkim.pid -p local:/var/run/opendkim/opendkim.sock
Restart=on-failure
ExecReload=/bin/kill -USR1 $MAINPID

[Install]
WantedBy=multi-user.target

to

[Unit]
Description=OpenDKIM DomainKeys Identified Mail (DKIM) Milter
Documentation=man:opendkim(8) man:opendkim.conf(5) man:opendkim-genkey(8) man:opendkim-genzone(8) man:opendkim-testadsp(8) man:opendkim-testkey http://www.opendkim.org/docs.html
After=network.target nss-lookup.target[Service]
Type=forking
PIDFile=/var/run/opendkim/opendkim.pid
User=opendkim
UMask=0007
ExecStart=/usr/sbin/opendkim -P /var/run/opendkim/opendkim.pid -p local:/var/run/opendkim/opendkim.sock -p inet:8891:localhost
Restart=on-failure
ExecReload=/bin/kill -USR1 $MAINPID

[Install]
WantedBy=multi-user.target

run (as root)

  • systemctl daemon-reload
  • /etc/init.d/opendkim restart

I hate systemd – that caused me six days of bug hunting it is limiting

Postfix needs a blog post on its own.

Opendmarc

I needed to re-enable it to start on boot oh the joys of systemd where init.d is thought as an unreliable forgetful moron and systemd knows best when clearly it is as fucked up (my blog)

It still did connect so it is a journey in systemd to fix (see opendkim magic above)

dmarc reports does not like interval and day together which appeared ok in Jessie

It is still a bit broken so nobody is being sent reports – not that many dmarc enabled domains who ‘specailise’ in just that really care about (my blog).  HistoryFile does not record data – why – no idea

-rw-rw-r– 1 opendmarc opendmarc 0 Jul 10 10:08 opendmarc.log

So a headscratcher. – and not something i can fix.

Postgres 9.4

I chowned a snakeoiil key – tested more cosmetic this than a issue which continues from Jessie..

Logwatch

Is a use full thing in my opinion although a little lacking in places moves from 7am to midnight for timing

Bind

Stops telling you if you do not have a specific spf record even though i have text records containing spf for the benefit of all the mostly retarded who run microsoft windows servers who have issues

mod_defensibile

Jury is out on if this is broken or the dns is bad. Or alternatively no rbl listed ip’s visited.

To fix

opendmarc loging, postfix startup, mod_defensible

Would i recommend the upgrade – at this point no.

Debian jessie to stretch server upgrade notes

I did basic workstations here (my blog) and there (my blog) note the networking issues which is also pertinent to servers Some workstation issues of help discovered early on where

virtual box

Needs help form the incompetent fools at oracle (see wiki) as the kernel modules are now outside the remit of debian support – speaks volumes of oracle.whom generally turn most things into a disaster like java (my blog).- can you wean me off virtualbox with some other manager suggestions welcome in comments.

systemd

needs a grub config line and a TTYVTDisallocat=no in otherwise you have no idea if it works or not.when loading you get some messages but without systemd config you will know about a few things see the wiki to set up [not hard]

The server entailed lots of backups and copies of old data all over the place just in case thing go wrong.

Day 1:

After backups change your sources file to stretch, update and apt -f full-update..2784 packages later (3 hours) i had a debian stretch os installed, cannot really call it a server though as fail2ban, postfix, imap and apache barely work.

dns,postgresql and ssh kind of worked though

Sisyphus is still a role model

I think upgrading from i386 to i686 caused the zoo a lot of issues, apt autoremove did not help and i had to remove over a two hundred remains of jessie packages.via aptitude.

Apache2 – cant do cgi and my cgi files did work in jessie

Fail2ban – honestly no idea what is happening here, deinstalled it

Postfix – missing loading four other postfix instances

Opendmarc – is mia

Cyrus – the jump from 2.4 to 2.5 means foo becomes foo_bar – your config files need massive changes, need to reconstuct databases too.   if you know what a DBERROR db5 is then your doing better than i am

Good news printing (cups) works and networking [see above link] survived.  I considered that a win

Day 2

With a fresh pair of eyes, i ‘fix’ crappy virtual box and discover to my delight that the zoo’s cron jobs still work.   I need to remove that trash for something better that does a virtual memory space when i test things.

Cyrus Imap

Gets weirder and werieder

  • it listens on http port 8008.(REALLY)
  • mboxlist and deliver commands seem to be not used
  • sasl logins are from the twilight  zone

I got a paired down (brand new config) that kind of works although three zoo domains cannot open the mail.

poledancing

Postfix

goes to version three expect to use one of these (not here)  If like the zoo you have more than more postfix instance then your need systemd to start it as the init.d scipt is dancing with the faeries and now only loads /etc/postfix.

spf

defaultseedonly becomes testonly – spf has to have some kind of issue and alert you about with debian upgrades or you never know your doing one.

Day 2 was kind of a success.  Even if the mail was flowing in via my actions rather than a systemd startup action and postfix and cyrus kind of work i think.

I began to feel that debain might work rather than simply tell me that estortic_command_lines may have changed.

Day 3:

sasl

Issues are caused by old sockets in cyrus.  Go to your imap mail store directory and the sockets directory and delere.  I did not have to delete *.db’s but even after a reconstruction its not explictly something advertised.   – I appear to be able to receive and process inbound mail

postfix systemd

With the magic of a console i started other instances of postfix and it appears to work my additional systemd scripts dont work one shows a bash shell and the others no bash shell – i hate systemd.- i might need postfix-multi but do not like the idea of it with my existing config,

Day 4

Good news first – amavis seems to work no issues, and now back to problems

apache cgi/mailgraph

I have weird apache error codes but not a meaning as to what they mean i think

  • ah000128 start
  • ah000169 restart
  • ah001909  ssl mismatch (warn)
  • ah002811 script alias issue ?
  • ah000094 ?

google searching for those is a miss they like 404 error codes  – cgi is well broken but that seems down to perl -i had to get rid of perl -wT and run perl -w so getting there.

Moving mailgraph.cgi to cgi-bin fixes the issue (we just need the images which are called via javascript url method).  I gave up /usr/lib/cgi-bin and did cgi-bin my own way.

tls/apache

appears to work unhindered like Jessie not a fuckup

fail2ban

Apparently does work – just reconfigure from scratch

Bind

version 9.10 apparently means it now do caa records without encoding, it has a geoip feature that it loves to advertise.

Conclusion

worst thing: cyrus imap

less worst thing perl ‘changes’ (cgi)

stuff to still fix

  • clean out etc old entries
  • postfix start up ‘issues’
  • remove on disk backups
  • opendmarc reporting is not working
  • check email sending with dkim (works locally)
  • postgresql 9.4 refuses to load but the 9,6 version means i do not have load it twice – a bit botched but progress

notable mentions to spf – good to see that i still had to change something.

Hope that has not put you off but that was my rather fraught upgrade experience.  Perhaps i should have gone from Jessie i386 to 686 and then to Stretch.

I can work on the issues at a more leisurely pace now

The pingbacks to this site below update on this post and resolve issues i had.

jessie to stretch debian upgrade

I completed another debian workstation update (my blog) this time it is an ancient laptop and after changing sources and apt -f full-upgrade it suffered no stops, i had to autoremove stuff to get x sessions working and the box works as far as i can tell by printing and browsing.

The console is bereft of information which can be fixed and despite some oddites with with sane config files not much to report on.  Hardware wise it has a crappy broadcom wifi adapter which now downloads (check your sources file) from a new domain name to me.

More boring than troublesome

So its now on to upgrading the zoo’s servers which is when the long days start since there not just clients.

debian virtual networking – four hours and four lines later

So i did this (my blog) and discovered my example network config did not transfer. Eventually i ended up with a config command of

allow-hotplug eth0:1

And it returned to working order, Debian 8 did not appear to need that.

Yep we all be muppets

A full example

auto eth0
allow-hotplug eth0
iface eth0 inet dhcp

auto lo
iface lo inet loopback

allow-hotplug eth0:1 #<needed per instance
iface eth0:1 inet static
address 10.0.0.1
netmask 255.0.0.0

I added back the the dhcp licence which the debian installer also removed. Oh also watch out for /etc/default/networking.

installing Debian Stretch on a laptop

a stretch you may know

Bananas knew that debian stretch (version 9 stable) was recently released.  I offered my laptop as a sacrifice victim  and although this is not a really complex debian install i got out of the temple of experimentation with few issues or wounds

The name is something i found associative with testing.so perhaps a odd name in that respect.

The install needed /etc/apt/sources to be changed from Jessie to Stretch and then apt update, and apt full upgrade.

I ran as root in the console [NO DISPLAY MANAGER] and got a few prompts for

  • /etc/issue
  • /etc/issue.net

Then systemd packed up the install (as usual systemd would do stupid shit) when it tried to unmount and remount the disk.

I used apt -f full update to get on after being told that was the Potterang way.

More prompts for

  • cups-browsed.conf
  • sddm and kdm ‘managers’

Do read the install notes on screen – networking has been significantly reworked which if like me your going to be doing server upgrades too needs a hairy eyeball.

Three hours later (our lan is rated for 100mb only*) i see

And it mostly works – i had to restart cups to test printing, I am writing this on the stretch os, I also have a caps lock indicator something i lost when debian testing went and died on me months ago.

The install petered out when a couple of server daemons i had not yet configured where deemed a bit too fucked up by the debian install.

Something i liked over debian testing was that startx seems to work rather than have to use sddm – x log files move from /var/log too.

It is early days and while this is a simple upgrade i do not find it step backward a niggle – redshift needs a better kde implementation for plasmoid.  –  At this level not a problem.

A warning networking in debian 9 does not allow for multiple addresses per network card so it may be the case that i have to find another linux os for the zoo’s server

I have mentioned most of the above items in my blog for context if you have no idea what sddm or something else is that is not my problem.

Now i click schedule.

*something i cannot be bothered to upgrade