We have been working with this one client for about 4 months on an OpenSSL issue. Today we received this awesome update on his system upgrades that I wanted to share with you. A system administrators job is never done, and what seems like a very straightforward task has twists and turns, dependencies and unintended discoveries.

To all the SysAdmins out there who keep software up to date and systems safe – thank you!

Hello fellas,

So, we are 4 months further and as of this weekend both mail.*****.com
and mail.********.com are (finally) supporting TLS 1.2. Here is an excerpt of
what I had to do to get there (for your education and entertainment).

See back in January both mail servers were running Redhat Enterprise Linux 5.
The direction Redhat took with RHEL 6 and 7 never suited my small office setup,
so I never felt the need to upgrade from what was a damn’ nice stable platform.
Who can complain about a server uptime of 628 days?

I had been evaluating various versions of Fedora Linux to see what would suit me
and had settled on Fedora 15, as it supported the latest Firefox browser as well
as the replacement of the (wretched nouveau) graphics driver with the nice work
by nVIDIA, essential for my all-3D-graphics crash analysis work. Getting that to
work on newer versions of Fedora is getting increasingly impossible.
I am not enthused by the KDE 4 Plasma Desktop as KDE 3 was nicer, quicker,
and more stable, but with gnome going the route of cell phone interface, there
is little choice.With all my workstations in use, I had to buy an extra system to install the newer OS.
Fedora 15 comes with openssl 1.0.0d, which does not support TLS 1.2
It can be upgraded to openssl 1.0.0j, but that doesn’t add any functionality.
Oh, and BTW you don’t want to upgrade to the “latest” kernel for Fedora 15, as it
breaks the system. Done that twice. What can I say, I learn slow.An openssl upgrade from source, therefore, was the only option.
As one guy on the interwebs advised: Installing a newer version of openssl
on top of your existing version will seriously break your operating system.

He wasn’t lying.
By my count there are 153 packages that use one or more openssl libraries.
I installed openssl-1.0.2f on the new system.
That immediately broke remote access as Redhat compiled openssh to only use a
fips enabled openssl, which carries a unique identifier.
ssh: OpenSSL version mismatch. Built against 10000003, you have 1000206f
The openssh source RPM also demands that the openssl library be fips enabled.
Of course the Fedora 15 fips-1.3 was too old to be used with openssl-1.0.2 which
requires fips-2.0Reset, start over.
First compile and install fips-2.0, then build openssl. By the time I got to that,
the newest release of openssl was 1.0.2h, so I went for that one. They change often!
A “manual” compile and install has the disadvantage that it throws off the RPM
database, which in turn precludes other yum/rpm upgrades as the required
packages, while installed, are not found in the database. Ergo, an rpmbuild is the
only way to go. The spec file that comes with openssl-1.0.2h was seriously
lacking. It was my first encounter with the rpm spec file format and I can now
say that I know everything about them, I never wanted to!
Nevertheless openssl-1.0.2h-fips became operational on my system in March.
This then allowed the rpmbuild of openssh, utilizing the new openssl-fips.Another serious bad-ass is php, which also needs to be recompiled after the
change of openssl versions. Moreover php must be compiled after Apache-httpd
is compiled. So, I did that one first. As I haven’t kept up with what the plans
are for Apache 2.4, I stuck with the newest httpd-2.2.31 I found two bugs, that
I fixed. No, nobody is interested in hearing about the bugs I fixed, because
nobody cares about the problems of running an abandoned version on an
outdated OS platform. So, I’ll be the only one in the world running httpd-2.2.35 😉 The only freebee was sendmail. Fedora 15 comes with sendmail-8.14, which
already has the capability to enforce TLS 1.2 and disable SSLv3, although there
is no macro support yet. But, hey, I always wanted to learn about the sendmail
configuration language anyways, right?! :-\

I did compile the latest version of dovecot to provide the pop and imap interface
to the mail server for my Thunderbird mail client.
Then I had a fight with saslauth, which I still don’t quite know what it does exactly,
but I do now know that the actual authentication is done through pam and that
unlike Redhat Enterprise Server, the default Fedora installation does not come with
pam configuration files for smtp, imap and pop.
There are only three lines to those files:
#%PAM-1.0
auth       include      password-auth
account    include      password-auth
But if you don’t have one for each service, it doesn’t work. What a great idea to
not have a fall-back default and also not write error messages to a log file! :-\
That was it for my first server, which took on mail.****.com March 29th.That freed-up the server that had been doing mail.*****.com to be upgraded
and configured to be mail.****.com Second time always goes quicker, right? Wrong!
First of all that workstation did not like being switch off and broke the system disk.
I had no idea how many “fortune seekers” are now operating on Ebay and got a
used SCSI disk with SCA interface shipped that had been advertised as a new disk
with 68-pin interface, twice! So we learn. But with shipping back and forth, that
wasted nearly a whole month.
In the meantime I learned everything I never wanted to know about certificates,
Certificate Authorities, certificate chains, cyphers, certificate sign requests, certificate
revocation lists, and OCSP responders. But I now have a wonderful set of certificates
in place. Still the root cert is self-signed, because I am a cheapskate! 🙂
That was also the system that taught me about the latest kernel for Fedora 15
kernel-2.6.43.8-1.rc1.fc15. being broken. 🙁   So, I’ll stick with kernel-2.6.38.6-26.rc1.fc15Anyways, finally this past weekend was the time of the big switcheroo. I had done all
the testing I could do, without actually throwing the switch at the router.
It was Sunday morning 00:36 when I changed the router settings to forward ports
586,109, and 52525 to “hork09” instead of “hork06”. It was immediately clear that
my Thunderbird email client, operating through the dovecot interface, could not
authenticate with the mail server. Since that had been working fine for ****.com
for nearly two months that was puzzling. But a quick comparison of the sendmail.mc
configuration files showed the reason why. This is how I had it:
dnl # PLAIN is the preferred plaintext authentication method and used by
dnl # Mozilla Mail and Evolution, though Outlook Express and other MUAs do
dnl # use LOGIN. Other mechanisms should be used if the connection is not
dnl # guaranteed secure.
dnl # Please remember that saslauthd needs to be running for AUTH.
dnl #
dnl TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
dnl define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
dnl #
dnl # Rudimentary information on creating certificates for sendmail TLS:
dnl #     cd /etc/pki/tls/certs; make sendmail.pem
dnl # Complete usage:
dnl #     make -C /etc/pki/tls/certs usage

I was supposed to have changed that to:

dnl # PLAIN is the preferred plaintext authentication method and used by
dnl # Mozilla Mail and Evolution, though Outlook Express and other MUAs do
dnl # use LOGIN. Other mechanisms should be used if the connection is not
dnl # guaranteed secure.
dnl # Please remember that saslauthd needs to be running for AUTH.
dnl #
TRUST_AUTH_MECH(`LOGIN PLAIN')dnl
define(`confAUTH_MECHANISMS', `LOGIN PLAIN')dnl
dnl #
dnl # Rudimentary information on creating certificates for sendmail TLS:
dnl #     cd /etc/pki/tls/certs; make sendmail.pem
dnl # Complete usage:
dnl #     make -C /etc/pki/tls/certs usage

But I had been so focused on that “Please remember that saslauthd needs to be
running for AUTH”, that I had been struggling with on *****.com, that I had
completely forgotten to uncomment and edit those two critical lines.
Yeah, you can laugh at me. I did! 🙂  Anyways, by 01:02 I had that little mishap
corrected and was sending out emails, left and right.
(mind you, outgoing emails for ****.com by default go through smtp.*****.org
as they have some policy about that).
Then there was the little thingy about receiving the same email over and over again.
That was solved by 2am when I changed the permissions of the /var/spool/mail
directory such that dovecot not only could retrieve the emails, but also could
remove them from the server in the process.

Of course at the moment-supreme I also changed the “TLS-mode” setting at DuoCircle
from “disabled” to “required”. Everybody happy! Now my login credentials are
properly secured as they travel from my upstairs office down to the mail server in
the basement as well as between that and the mailhop server on the other side of
the country.
With the constant flood of attacks I see on my web and mail servers, it is surprising
those credentials weren’t intercepted and misused these past 5 months.
Maybe it be a good idea if I changed my mailhop password, but then I need to update
the mail/access file too and restart sendmail. Maybe next Sunday..? yeah!

I think “we” can close this ticket now! 🙂
Cheers,

Pin It on Pinterest

Share This