--- title: "Trouble Ticket Hall of Fame OpenSSL | DuoCircle" description: "We have been working with this one client for about 4 months on an OpenSSL issue." image: "https://www.duocircle.com/images/og-default.png" canonical: "https://www.duocircle.com/blog/announcements/trouble-ticket-hall-fame-openssl/" --- Quick Answer A four-month support ticket walking through one customer's TLS 1.2 upgrade across two mail servers running Fedora 15 (chosen for legacy graphics-driver and KDE 3 support). The path: Fedora 15 shipped OpenSSL 1.0.0d (no TLS 1.2), upgrading in-place broke 153 dependent packages, so a clean rebuild was required. FIPS-2.0 had to be compiled before OpenSSL 1.0.2h, then OpenSSH rebuilt against the new FIPS-enabled OpenSSL, then Apache httpd 2.2.31 and PHP recompiled in order. Sendmail 8.14 supported TLS 1.2 directly, dovecot was rebuilt for IMAP and POP, and saslauthd PAM configs for smtp, imap, and pop had to be created manually since Fedora did not ship them by default. A failed eBay disk and a broken kernel update added a month of delay. Final cutover at 00:36 Sunday hit a forgotten saslauthd configuration line; corrected by 01:02\. TLS-mode at DuoCircle flipped from disabled to required. Share [ ](https://www.linkedin.com/sharing/share-offsite/?url=undefined%2Fblog%2Fannouncements%2Ftrouble-ticket-hall-fame-openssl%2F "Share on LinkedIn") [ ](https://twitter.com/intent/tweet?text=Trouble%20Ticket%20Hall%20of%20Fame%20OpenSSL&url=undefined%2Fblog%2Fannouncements%2Ftrouble-ticket-hall-fame-openssl%2F "Share on X/Twitter") [ ](https://www.facebook.com/sharer/sharer.php?u=undefined%2Fblog%2Fannouncements%2Ftrouble-ticket-hall-fame-openssl%2F "Share on Facebook") [ ](https://reddit.com/submit?url=undefined%2Fblog%2Fannouncements%2Ftrouble-ticket-hall-fame-openssl%2F&title=Trouble%20Ticket%20Hall%20of%20Fame%20OpenSSL "Share on Reddit") [ ](mailto:?subject=Trouble%20Ticket%20Hall%20of%20Fame%20OpenSSL&body=Check out this article: undefined%2Fblog%2Fannouncements%2Ftrouble-ticket-hall-fame-openssl%2F "Share via Email") ![OpenSSL](https://media.mailhop.org/duocircle/images/2016/06/anti-phishing-software-2237.jpg) We have been working with this one client for about 4 months on an OpenSSL issue. Today we received this awesome update on his system upgrades that I wanted to share with you. A system administrators job is never done, and what seems like a very straightforward task has twists and turns, dependencies and unintended discoveries. To all the SysAdmins out there who keep software up to date and systems safe, thank you! Hello fellas, So, we are 4 months further and as of this weekend both mail.\*\*\*\*\*.com and mail.\*\*\*\*\*\*\*\*.com are (finally) supporting TLS 1.2\. Here is an excerpt of what I had to do to get there (for your education and entertainment). See back in January both mail servers were running Redhat Enterprise Linux 5. The direction Redhat took with RHEL 6 and 7 never suited my small office setup, so I never felt the need to upgrade from what was a damn’ nice stable platform. Who can complain about a server uptime of 628 days? I had been evaluating various versions of Fedora Linux to see what would suit me and had settled on Fedora 15, as it supported the latest Firefox browser as well as the replacement of the (wretched nouveau) graphics driver with the nice work by n**VIDIA**, essential for my all-3D-graphics crash analysis work. Getting that to work on newer versions of Fedora is getting increasingly impossible. I am not enthused by the KDE 4 Plasma Desktop as KDE 3 was nicer, quicker, and more stable, but with gnome going the route of cell phone interface, there is little choice.With all my workstations in use, I had to buy an extra system to install the newer OS. Fedora 15 comes with openssl 1.0.0d, which does not support TLS 1.2 It can be upgraded to openssl 1.0.0j, but that doesn’t add any functionality. Oh, and BTW you don’t want to upgrade to the “latest” kernel for Fedora 15, as it breaks the system. Done that twice. What can I say, I learn slow.An openssl upgrade from source, therefore, was the only option. As one guy on the interwebs advised: Installing a newer version of openssl on top of your existing version will seriously break your operating system. He wasn’t lying. By my count there are 153 packages that use one or more openssl libraries. I installed openssl-1.0.2f on the new system. That immediately broke remote access as Redhat compiled openssh to only use a fips enabled openssl, which carries a unique identifier. ssh: OpenSSL version mismatch. Built against 10000003, you have 1000206f The openssh source RPM also demands that the openssl library be fips enabled. Of course the Fedora 15 fips-1.3 was too old to be used with openssl-1.0.2 which requires fips-2.0Reset, start over. First compile and install fips-2.0, _then_ build openssl. By the time I got to that, the newest release of openssl was 1.0.2h, so I went for that one. They change often! A “manual” compile and install has the disadvantage that it throws off the RPM database, which in turn precludes other yum/rpm upgrades as the required packages, while installed, are not found in the database. Ergo, an rpmbuild is the only way to go. The spec file that comes with openssl-1.0.2h was seriously lacking. It was my first encounter with the rpm spec file format and I can now say that I know everything about them, I never wanted to! Nevertheless openssl-1.0.2h-fips became operational on my system in March. This then allowed the rpmbuild of openssh, utilizing the new openssl-fips.Another serious bad-ass is php, which also needs to be recompiled after the change of openssl versions. Moreover php must be compiled after Apache-httpd is compiled. So, I did that one first. As I haven’t kept up with what the plans are for Apache 2.4, I stuck with the newest httpd-2.2.31 I found two bugs, that I fixed. No, nobody is interested in hearing about the bugs I fixed, because nobody cares about the problems of running an abandoned version on an outdated OS platform. So, I’ll be the only one in the world running httpd-2.2.35 😉 The only freebee was sendmail. Fedora 15 comes with sendmail-8.14, which already has the capability to enforce TLS 1.2 and disable SSLv3, although there is no macro support yet. But, hey, I always wanted to learn about the sendmail configuration language anyways, right?! :-\\ I did compile the latest version of dovecot to provide the pop and imap interface to the mail server for my Thunderbird mail client. Then I had a fight with saslauth, which I still don’t quite know what it does exactly, but I do now know that the actual authentication is done through pam and that unlike Redhat Enterprise Server, the default Fedora installation does not come with pam configuration files for smtp, imap and pop. There are only three lines to those files: #%PAM-1.0 auth include password-auth account include password-auth But if you don’t have one for each service, it doesn’t work. What a great idea to _not_ have a fall-back default and also _not_ write error messages to a log file! :-\\ That was _it_ for my first server, which took on mail.\*\*\*\*.com March 29th.That freed-up the server that had been doing mail.\*\*\*\*\*.com to be upgraded and configured to be mail.\*\*\*\*.com Second time always goes quicker, right? Wrong! First of all that workstation did not like being switch off and broke the system disk. I had no idea how many “fortune seekers” are now operating on Ebay and got a used SCSI disk with SCA interface shipped that had been advertised as a new disk with 68-pin interface, twice! So we learn. But with shipping back and forth, that wasted nearly a whole month. In the meantime I learned everything I never wanted to know about certificates, Certificate Authorities, certificate chains, cyphers, certificate sign requests, certificate revocation lists, and OCSP responders. But I now have a wonderful set of certificates in place. Still the root cert is self-signed, because I am a cheapskate! 🙂 That was also the system that taught me about the latest kernel for Fedora 15 kernel-2.6.43.8-1.rc1.fc15\. being broken. 🙁 So, I’ll stick with kernel-2.6.38.6-26.rc1.fc15Anyways, finally this past weekend was the time of the big switcheroo. I had done all the testing I could do, without actually throwing the switch at the router. It was Sunday morning 00:36 when I changed the router settings to forward ports 586,109, and 52525 to “hork09” instead of “hork06”. It was immediately clear that my Thunderbird email client, operating through the dovecot interface, could not authenticate with the mail server. Since that had been working fine for \*\*\*\*.com for nearly two months that was puzzling. But a quick comparison of the sendmail.mc configuration files showed the reason why. This is how I had it: | dnl # PLAIN is the preferred plaintext authentication method and used by dnl # Mozilla Mail and Evolution, though Outlook Express and other MUAs do dnl # use LOGIN. Other mechanisms should be used if the connection is not dnl # guaranteed secure. dnl # Please remember that saslauthd needs to be running for AUTH. dnl # dnl TRUST\_AUTH\_MECH(EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl dnl define(confAUTH\_MECHANISMS’, \`EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN’)dnl dnl # dnl # Rudimentary information on creating certificates for sendmail TLS: dnl # cd /etc/pki/tls/certs; make sendmail.pem dnl # Complete usage: dnl # make -C /etc/pki/tls/certs usage | | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | I was supposed to have changed that to: | dnl # PLAIN is the preferred plaintext authentication method and used by dnl # Mozilla Mail and Evolution, though Outlook Express and other MUAs do dnl # use LOGIN. Other mechanisms should be used if the connection is not dnl # guaranteed secure. dnl # Please remember that saslauthd needs to be running for AUTH. dnl # TRUST\_AUTH\_MECH(LOGIN PLAIN')dnl define(confAUTH\_MECHANISMS’, \`LOGIN PLAIN’)dnl dnl # dnl # Rudimentary information on creating certificates for sendmail TLS: dnl # cd /etc/pki/tls/certs; make sendmail.pem dnl # Complete usage: dnl # make -C /etc/pki/tls/certs usage | | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | But I had been _so_ focused on that “Please _remember_ that saslauthd needs to be running for AUTH”, that I had been struggling with on \*\*\*\*\*.com, that I had completely forgotten to uncomment and edit those two critical lines. Yeah, you can laugh at me. I did! 🙂 Anyways, by 01:02 I had that little mishap corrected and was sending out emails, left and right. (mind you, outgoing emails for \*\*\*\*.com by default go through smtp.\*\*\*\*\*.org as they have some policy about that). Then there was the little thingy about receiving the same email over and over again. That was solved by 2am when I changed the permissions of the /var/spool/mail directory such that dovecot not only could retrieve the emails, but also could remove them from the server in the process. Of course at the moment-supreme I also changed the “TLS-mode” setting at DuoCircle from “disabled” to “required”. Everybody happy! Now my login credentials are properly secured as they travel from my upstairs office down to the mail server in the basement as well as between that and the mailhop server on the other side of the country. With the constant flood of attacks I see on my web and mail servers, it is surprising those credentials weren’t intercepted and misused these past 5 months. Maybe it be a good idea if I changed my mailhop password, but then I need to update the mail/access file too and restart sendmail. Maybe next Sunday..? yeah! I think “we” can close this ticket now! 🙂 Cheers, ![Brad Slavin](https://media.mailhop.org/dmarcreport/images/team/brad-slavin.jpg) Brad Slavin General Manager General Manager at DuoCircle. Product strategy and commercial lead across the email security portfolio. ## Secure your email infrastructure Protect, authenticate, and deliver. Contact our team to find the right solution. [Contact Sales](/contact/) [Explore Products](/products/) ## Related Articles [ News 3m Alert: Fix SPF & DKIM Settings For Your Email Forwarding Set Up Through Microsoft o365 SMTP Server Or Your Emails May End Up In Spam Jul 20, 2021 ](/blog/announcements/alert-fix-spf-dkim-settings-for-your-email-forwarding-set-up-through-microsoft-o365-smtp-server-or-your-emails-may-end-up-in-spam/)[ News 1m April Spam Filtering Uptime Report May 4, 2016 ](/blog/announcements/april-spam-filtering-uptime-report/)[ News 2m Changes to Spam Filtering Technology Feb 8, 2023 ](/blog/announcements/changes-to-spam-filtering-technology/)[ News 4m Cyber Security News Update, Week 1 of 2020 Jan 3, 2020 ](/blog/announcements/cyber-security-news-update-week-1-of-2020/) ```json {"@context":"https://schema.org","@type":"Organization","name":"DuoCircle LLC","url":"https://www.duocircle.com","logo":{"@type":"ImageObject","url":"https://www.duocircle.com/images/duocircle-logo.png"},"description":"DuoCircle is a portfolio of specialized email products covering protection, authentication, delivery, and routing. We deliver about 90% of category-leader capability at roughly half the price, backed by experts who own the outcome. Trusted by 50,000+ organizations since 2014.","subOrganization":[{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138898167","name":"DMARC Report","url":"https://dmarcreport.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897474","name":"AutoSPF","url":"https://autospf.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897912","name":"Phish Protection","url":"https://www.phishprotection.com"}],"sameAs":["https://www.linkedin.com/company/duocircle","https://x.com/duocirclellc","https://www.facebook.com/duocirclellc","https://www.g2.com/products/phish-protection-by-duocircle/reviews","https://github.com/duocircle","https://www.crunchbase.com/organization/duocircle-llc"],"contactPoint":{"@type":"ContactPoint","contactType":"customer support","url":"https://support.duocircle.com"},"knowsAbout":["Email Security","Email Authentication","SPF","DKIM","DMARC","Phishing Protection","Spam Filtering","SMTP Relay","Email Deliverability","Email Forwarding"]} ``` ```json {"@context":"https://schema.org","@type":"WebSite","name":"DuoCircle LLC","url":"https://www.duocircle.com","description":"DuoCircle is a portfolio of specialized email products covering protection, authentication, delivery, and routing. We deliver about 90% of category-leader capability at roughly half the price, backed by experts who own the outcome. Trusted by 50,000+ organizations since 2014.","publisher":{"@type":"Organization","name":"DuoCircle LLC","url":"https://www.duocircle.com","logo":{"@type":"ImageObject","url":"https://www.duocircle.com/images/duocircle-logo.png"},"description":"DuoCircle is a portfolio of specialized email products covering protection, authentication, delivery, and routing. We deliver about 90% of category-leader capability at roughly half the price, backed by experts who own the outcome. Trusted by 50,000+ organizations since 2014.","subOrganization":[{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138898167","name":"DMARC Report","url":"https://dmarcreport.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897474","name":"AutoSPF","url":"https://autospf.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897912","name":"Phish Protection","url":"https://www.phishprotection.com"}]}} ``` ```json [{"@context":"https://schema.org","@type":"BlogPosting","headline":"Trouble Ticket Hall of Fame OpenSSL","description":"We have been working with this one client for about 4 months on an OpenSSL issue.","url":"https://www.duocircle.com/blog/announcements/trouble-ticket-hall-fame-openssl/","datePublished":"2016-06-01T23:41:42.000Z","dateModified":"2025-05-09T16:46:56.000Z","dateCreated":"2016-06-01T23:41:42.000Z","author":{"@type":"Person","@id":"https://www.duocircle.com/authors/brad-slavin/#person","name":"Brad Slavin","url":"https://www.duocircle.com/authors/brad-slavin/","jobTitle":"General Manager","description":"Brad Slavin runs DuoCircle, the company behind DMARC Report, AutoSPF, Phish Protection, and Mailhop. His focus is product strategy, customer relationships, and the commercial and compliance side of email authentication (DPAs, SLAs, enterprise procurement).","image":"https://media.mailhop.org/dmarcreport/images/team/brad-slavin.jpg","knowsAbout":["Email Security Strategy","SaaS Product Management","Enterprise Compliance","Customer Success","Email Deliverability Business"],"worksFor":{"@type":"Organization","name":"DuoCircle LLC","url":"https://www.duocircle.com"},"sameAs":["https://www.linkedin.com/in/bradslavin"]},"publisher":{"@type":"Organization","name":"DuoCircle LLC","url":"https://www.duocircle.com","logo":{"@type":"ImageObject","url":"https://www.duocircle.com/images/duocircle-logo.png"},"description":"DuoCircle is a portfolio of specialized email products covering protection, authentication, delivery, and routing. We deliver about 90% of category-leader capability at roughly half the price, backed by experts who own the outcome. Trusted by 50,000+ organizations since 2014.","subOrganization":[{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138898167","name":"DMARC Report","url":"https://dmarcreport.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897474","name":"AutoSPF","url":"https://autospf.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897912","name":"Phish Protection","url":"https://www.phishprotection.com"}]},"mainEntityOfPage":{"@type":"WebPage","@id":"https://www.duocircle.com/blog/announcements/trouble-ticket-hall-fame-openssl/"},"articleSection":"announcements","keywords":"","wordCount":1586,"image":{"@type":"ImageObject","url":"https://media.mailhop.org/duocircle/images/2016/06/anti-phishing-software-2237.jpg","caption":"OpenSSL","width":900,"height":600},"speakable":{"@type":"SpeakableSpecification","cssSelector":[".answer-block","h1"]}},{"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog","item":"https://www.duocircle.com/blog/"},{"@type":"ListItem","position":2,"name":"News"},{"@type":"ListItem","position":3,"name":"Trouble Ticket Hall of Fame OpenSSL","item":"https://www.duocircle.com/blog/announcements/trouble-ticket-hall-fame-openssl/"}]}] ``` ```json {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://www.duocircle.com/"},{"@type":"ListItem","position":2,"name":"Blog","item":"https://www.duocircle.com/blog/"},{"@type":"ListItem","position":3,"name":"News","item":"https://www.duocircle.comundefined"},{"@type":"ListItem","position":4,"name":"Trouble Ticket Hall of Fame OpenSSL","item":"https://www.duocircle.com/blog/announcements/trouble-ticket-hall-fame-openssl/"}]} ``` ```json {"@context":"https://schema.org","@type":"BlogPosting","headline":"Trouble Ticket Hall of Fame OpenSSL","description":"We have been working with this one client for about 4 months on an OpenSSL issue.","url":"https://www.duocircle.com/blog/announcements/trouble-ticket-hall-fame-openssl/","datePublished":"2016-06-01T23:41:42.000Z","dateModified":"2025-05-09T16:46:56.000Z","dateCreated":"2016-06-01T23:41:42.000Z","author":{"@type":"Person","@id":"https://www.duocircle.com/authors/brad-slavin/#person","name":"Brad Slavin","url":"https://www.duocircle.com/authors/brad-slavin/","jobTitle":"General Manager","description":"Brad Slavin runs DuoCircle, the company behind DMARC Report, AutoSPF, Phish Protection, and Mailhop. His focus is product strategy, customer relationships, and the commercial and compliance side of email authentication (DPAs, SLAs, enterprise procurement).","image":"https://media.mailhop.org/dmarcreport/images/team/brad-slavin.jpg","knowsAbout":["Email Security Strategy","SaaS Product Management","Enterprise Compliance","Customer Success","Email Deliverability Business"],"worksFor":{"@type":"Organization","name":"DuoCircle LLC","url":"https://www.duocircle.com"},"sameAs":["https://www.linkedin.com/in/bradslavin"]},"publisher":{"@type":"Organization","name":"DuoCircle LLC","url":"https://www.duocircle.com","logo":{"@type":"ImageObject","url":"https://www.duocircle.com/images/duocircle-logo.png"},"description":"DuoCircle is a portfolio of specialized email products covering protection, authentication, delivery, and routing. We deliver about 90% of category-leader capability at roughly half the price, backed by experts who own the outcome. Trusted by 50,000+ organizations since 2014.","subOrganization":[{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138898167","name":"DMARC Report","url":"https://dmarcreport.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897474","name":"AutoSPF","url":"https://autospf.com"},{"@type":"Organization","@id":"https://www.wikidata.org/wiki/Q138897912","name":"Phish Protection","url":"https://www.phishprotection.com"}]},"mainEntityOfPage":{"@type":"WebPage","@id":"https://www.duocircle.com/blog/announcements/trouble-ticket-hall-fame-openssl/"},"articleSection":"announcements","keywords":"","wordCount":1586,"image":{"@type":"ImageObject","url":"https://media.mailhop.org/duocircle/images/2016/06/anti-phishing-software-2237.jpg","caption":"OpenSSL","width":900,"height":600},"speakable":{"@type":"SpeakableSpecification","cssSelector":[".answer-block","h1"]}} ```