kibana nginx proxy

server {
listen 80;
server_name kibana;

error_log /var/log/nginx/kibana.error.log;
access_log /var/log/nginx/kibana.access.log;

 

location / {
rewrite ^/(.*) /$1 break;
proxy_ignore_client_abort on;
proxy_pass http://localhost:5601;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;

}
}

VM online resize

No need to rescue / reboot or anything.

yum install cloud-utils-growpart

growpart /dev/sda 1

resize2fs /dev/sda1
resize2fs 1.46.5 (30-Dec-2021)
Filesystem at /dev/sda1 is mounted on /; on-line resizing required
old_desc_blocks = 16, new_desc_blocks = 28
The filesystem on /dev/sda1 is now 58719995 (4k) blocks long.

Dealing with igb nic driver dropping network

The problem looks like the device ‘disappears’ from the bus, and becomes inaccessible to the driver. If it happens early – the driver will not load, if it happens later – it may fail with sporadic access errors

cat /sys/module/pcie_aspm/parameters/policy
grubby --update-kernel ALL --args "pcie_aspm.policy=performance pcie_port_pm=off pcie_aspm=off"
dracut -f
ethtool -K eno1 tso off gso off gro off

 

cat /root/resetnic.sh

#!/bin/bash

gg_intel() {
journalctl -f | while IFS= read -r line; do
if echo “$line” | grep -q “Failed to read reg 0xc030!”; then

echo 1 > “/sys/bus/pci/devices/0000:07:00.0/remove”
echo 1 > “/sys/bus/pci/rescan”

echo “[$(date)] NIC Was reset!” >>/root/resetnic.log
fi
done
}
gg_intel

cat /etc/systemd/system/resetnic.service

[Unit]
Description=Reset NIC Service
After=network.target

[Service]
Type=simple
ExecStart=/bin/bash /root/resetnic.sh

[Install]
WantedBy=multi-user.target

systemctl daemon-reload

sudo systemctl enable resetnic.service sudo systemctl start resetnic.service

Set Up Automatic Update (Unattended Upgrades) on Ubuntu + Kernel reboots

Configure Automatic Security Update (Unattended Upgrades) on Ubuntu Server

First, install the unattended-upgrades package.

sudo apt update

sudo apt install unattended-upgrades

You need to install the update-notifier-common package in order to set up automatic reboot.

sudo apt install update-notifier-common

Then edit the 50unattended-upgrades file.

sudo nano /etc/apt/apt.conf.d/50unattended-upgrades

In this file, you can configure what packages should be automatically updated. By default, only security updates will be automatically installed, as indicated by the following lines. So there’s no need to change this section.

Unattended-Upgrade::Allowed-Origins {
"${distro_id}:${distro_codename}";
"${distro_id}:${distro_codename}-security";
"${distro_id}ESMApps:${distro_codename}-apps-security";
"${distro_id}ESM:${distro_codename}-infra-security";
"${distro_id}:${distro_codename}-updates";
"${distro_id}:${distro_codename}-proposed";
"${distro_id}:${distro_codename}-backports";
};

Unattended-Upgrade::Package-Blacklist {
};

Unattended-Upgrade::DevRelease "auto";
Unattended-Upgrade::AutoFixInterruptedDpkg "true";
Unattended-Upgrade::MinimalSteps "true";
Unattended-Upgrade::Remove-Unused-Kernel-Packages "true";
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
Unattended-Upgrade::Remove-Unused-Dependencies "true";
Unattended-Upgrade::Automatic-Reboot "true";
Unattended-Upgrade::Automatic-Reboot-WithUsers "true";
Unattended-Upgrade::Automatic-Reboot-Time "02:00";



 

 

Enable Automatic Security Update

Now that automatic security update is configured, we need to enable it by creating the 20auto-upgrades file.

sudo nano /etc/apt/apt.conf.d/20auto-upgrades

Copy and paste the following two lines into the file.

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Unattended-Upgrade "1";

 

Run Unattended Upgrade at a Specific Time

Unattended upgrade is run randomly between 12 AM to 7AM, so as to prevent load spike to mirror servers due to everyone running updates at the same time. You can manually run unattended upgrade with:

sudo unattended-upgrade -v

You can also add this command to your Cron job.

sudo crontab -e

Add the following line at the bottom of your Crontab file, so the unattended upgrade will run every day at 2 AM.

0 2 * * * sudo /usr/bin/unattended-upgrade -v

Check Restart

The checkrestart command can help you find out which processes need to be restarted after an upgrade. It is available from debian-goodies package.

sudo apt install debian-goodies

sudo checkrestart

 

 

Quotes

“If you own a machine, you are in turn owned by it, and spend your time serving it…” — Marion Zimmer Bradley, _The Forbidden Tower_

Fighting for peace is like fucking for virginity

 

Don’t steal; thou’lt never thus compete successfully in business. Cheat. — Ambrose Bierce

DNSBL: how does it work, and how to create and build an own

DNSBL is a DNS based [DNS = Domain Name System] blackhole list, which can be used as countermeasure against unwanted mail spam.

One of the most efficient ways to block mail spam is to do it on mail server level by denying incoming connects from spam sources, where the contacting remote machine is identified by its IP address which is checked against one or more DNSBLs on the fly.

The first DNSBL was the MAPS RBL (Realtime Blackhole List) by Paul Vixie in 1997. Since MAPS went commercial and RBL is now a service mark of MAPS LLC, the generic term DNSBL is preferred over RBL or DNSRBL in this article.

 

How does a DNSBL work

The mailserver checks the IP address of every incoming SMTP connect request against one or more DNSBLs = DNS based black lists.
These BLs are just a list of spam-related IP addresses, implemented as a DNS zone file, so that the list can be easily queried by simple and fast DNS lookups, and hence the name DNSBL.

The “positives” entries in that zone file (following example in BIND format) are A records and TXT records like

44.3.200.10.dnsbl.example.com.  IN A  127.0.0.3
                                IN TXT "spam source, rot in hell"
45.3.200.10.dnsbl.example.com.  IN A  127.0.0.3
                                IN TXT "spam source, rot in hell"
*.5.222.10.dnsbl.example.com.   IN A  127.0.0.10
                                IN TXT "confirmed DUL range, please use your ISP's smart mail host"
2.0.0.127.dnsbl.example.com.    IN A  127.0.0.2
                                IN TXT "My-private-blacklist: test record. The list is active"

Note: all domain names and/or IP addresses given in this arcticle are just examples and unrelated to the real world.

Please note the reversed notation of the IP address octets (ip4r) as the address is translated into an articifical subdomain name. So IP address 10.200.3.44 gets listed as 44.3.200.10.dnsbl.example.com.

The 127.0.0.x (the x to be greater than 1) is freely choosen by the zone maintainer. Entries may be grouped in such way that different 127.0.0.x may designate different classes of entries like spam sources, open relays, spam support sites, DULs, etc.
However, most mail server programs cannot see the difference, so it doesn’t really matter for the simple decision if we will accept a connection (= no entry) or not (= 127.0.0.x).

For testing purposes to check if the list is active and working, a 127.0.0.2 test record should be there per convention.

If our mail server gets a SMTP connect from remote IP address 10.200.3.44 and you configured the server do check the dnsbl.example.com DNSBL, the mail server now does a DNS lookup for the name 44.3.200.10.dnsbl.example.com (note the reversed notation of the IP address octets (ip4r)) and will get the following information from the DNS:
Address: 127.0.0.3
44.3.200.10.dnsbl.example.com text = "spam source, rot in hell"
Per convention, if the DNSBL returns a 127.0.0.x (for x > 1), it is a positive hit.
So the mail server now concludes from the positive hit “127.0.0.3” that the connecting machine is unwanted and may terminate the SMTP conversation by sending a final reject message consisting of an appropriate 5xx error code and the TXT information from the DNS query:
"553 spam source, rot in hell".
No mail seen at all.

Next our mail server gets a connect from remote IP address 10.222.5.99. A DNS lookup for the name 99.5.222.10.dnsbl.example.com will return the following information from the DNS:
Address: 127.0.0.10
99.5.222.10.dnsbl.example.com text = "confirmed DUL range, please use your ISP's smart mail host"
So the mail server will see a positive hit and consequently sending a 5xx reject message
"553 confirmed DUL range, please use your own ISP's mail host"
to terminate the SMTP conversation. No mail seen at all.

Next our mail server gets a connect from remote IP address 10.111.1.22. A DNS lookup for the name 22.1.111.10.dnsbl.example.com will return the following information from the DNS:
can't find 22.1.111.10.dnsbl.example.com. : NXDOMAIN
As there is no hit, the mail server will continue the SMTP dialogue with the remote server and accept the incoming mail.
Everything is fine here.

 

Why an own DNSBL ?

There are several public blackhole lists available, so you should NOT build an own one, unless you really need to.

Several specialized blocklists are listing open relays, SOCKS proxies, known spam sources, ISP dial up links (DUL), and others. Some wellknown DNSBLs are sbl.spamhaus.orgbl.spamcop.netlist.dsbl.orgrelays.ordb.orgdnsbl.sorbs.netdnsbl.njabl.org, just to name a few. There are many others.

More Links:
dmoz.org – DNSBL-listings.
openrbl.org – Check an IP address against common DNSBLs.
moensted.dk – Check an IP address against a long list of DNSBLs.
declude.com – Long list of IP based DNSBLs.

One need for an own BL could be, that you need an additional own private list to additionally check some “special” private entries not (yet) covered in the public BLs, or if you need an own private whitelist to be checked before the public blacklists. You should, however, implement a private list in a way that it does not leak out into the public due to DNS mis-configurations.

If you really should think that the world needs a new public BL, please re-consider several times, if you are prepared for both the organizational as well as the technical aspects of: (automatic) management of the BL, setting up a database for the list entries and batch procedures to automatically generate zone files from the database content, rules and processes (both manual and automatic) for listing, de-listing and expiring of entries, discussion and legal trouble with owners of listed IPs demanding de-listing against your rules, costs for the operation of several parallel servers to cover lookup demand load peaks and denial-of-service (DOS) attacks, funds for the resulting data traffic bandwidth costs for normal operation as well as excessive DOS-attacks, trouble with your network provider, who perhaps doesn’t want to attract DOS situations within his network and will kick you because he doesn’t want any trouble at all.

 

Howto: build an own DNSBL

For a DNSBL you need — as the name already implies — a DNS server. Depending on your environment, there are several ways to go.

The best way to build a BL would be to do it on a separate server with no interference to other DNS.
If you don’t yet have a DNS server running on the server you want to implement your own BL, then you may choose between several possibilities to set up a small own DNS especially dedicated for that purpose. You can choose between a fullblown BIND as well as other software like rbldns as part of djbdns, or rbldnsd.
Since my DNS runs under BIND, I don’t have own experience with the rbldns or rbldnsd and can’t tell you more about these.

If you only have one single server, and there already is a DNS server running, then you have basically 2 options. [As the DNS service is already running on port 53, you can’t have another DNS on the same machine on the same default port.] You can:
(1) adapt to this situation and implement the BL as an additional zone within that DNS you already have, fitting into its given setup.
(2) implement an additional DNS working on another private port like 530, and define a zone on the already running main DNS with “type forward;” pointing to the additional server.
I have choosen the 1st option, which will be described in the following chapter.
Caution: be careful to not screw up your setup. Only a small private DNSBL may coexist with the current load of your DNS. BIND may have performance problems with large zone files with many entries. The rbldns is said to handle larger zone files much better than bind. Don’t risk or kill your main DNS performance with just an additional BL.
You have been warned and continue in the next chapter on your own risk.

 

Add a DNSBL on a working BIND server

For a small domain site just being hosted on a single server, with SMTP mail and a primary authoritative BIND DNS serving just the site’s zone on the same box, here is the easy way to just add an additional small private low-traffic DNSBL on that same server box.
The small zone file with the spammy addresses is just plainly written with an ASCII editor, doing the ip4r reversing of the addresses manually. No database or other software is needed.

In the following example configuration, the main domain is example.com,
the DNSBL is dnsbl.example.com,
all is running on the same server box with IP addess 192.168.177.22.
Just do a search-and-replace to change to your own needs.

Step 0: Before you start: check your existing DNS setup for “example.com” with www.dnsreport.com and ensure that everything is OK and there are no current errors in your DNS setup. Create backup copies of the config and zone files, so that you could easily switch back, if would find yourself hosed later.

Step 1: Change your bind (named) config file /etc/named.conf to include the following:

options {
        ....
        # globally deny public lookups and public recursion
        allow-query { localhost; };
        allow-recursion { localhost; };
}

# main zone file for the domain "example.com"
zone "example.com" in {
        # we are the primary and authoritative server
        type master;
        file "example-com.zone";
        notify yes;
        # now allow public lookups just for this domain
        allow-query { any; };
        # allow zone transfer to your secondary nameservers only
        allow-transfer {
          xx.xx.xx.xx; yy.yy.yy.yy;
        };
};

# private block-list in a delegated sub-domain
zone "dnsbl.example.com" in {
        type master;
        file "dnsbl-example-com.zone";
        # we already had set this in the global options, but just to be sure again:
        #    no public queries to our DNSBL !!!
        allow-query { localhost; };
        # don't know if the following is required, 
        #    but it is perhaps not bad to definitely restrict
        allow-transfer { 127.0.0.1; };
};

Step 2: Change your domain’s primary zone file example-com.zone to include or look like the following:

$TTL 86400      ; 1 day
@               IN SOA  ns1 hostmaster (
                        2004032201 ; serial
                        7200       ; refresh (2 hours)
                        5400       ; retry (1.5 hours)
                        1814400    ; expire (3 weeks)
                        86400      ; minimum (1 day)
                        )
                IN NS   ns1
                IN NS   ns2.secondary-dns.xx.
                IN MX   10 mail
                IN MX   20 mail.other-mx.yyy.
;
                IN A    192.168.177.22
;
ns1             IN A    192.168.177.22
dnsbl           IN NS   ns1
ftp             IN A    192.168.177.22
mail            IN A    192.168.177.22
www             IN A    192.168.177.22
;

The “@” imports the domain name. Everything is on the same server except the secondary nameserver. Important is the delegation of the dnsbl to an own nameserver "dnsbl IN NS ns1", even if it is the same.

Step 3: Create/modify your DNSBL zone file dnsbl-example-com.zone as follows:

$TTL 86400      ; 1 day
@               IN SOA  ns1.example.com. hostmaster.example.com. (
                        2004032201 ; serial
                        7200       ; refresh (2 hours)
                        5400       ; retry (1.5 hours)
                        1814400    ; expire (3 weeks)
                        86400      ; minimum (1 day)
                        )
                IN NS   ns1.example.com.
;
                IN A    192.168.177.22
;
; -----------------------------------------------------------------
2.0.0.127       IN A    127.0.0.2
                IN TXT  "example.com test record"
3.0.0.127       IN A    127.0.0.3
                IN TXT  "example.com verified spam source"
10.0.0.127      IN A    127.0.0.10
                IN TXT  "example.com confirmed DUL range"
;
; -----------------------------------------------------------------
44.3.200.10     IN A  127.0.0.3
                IN TXT "spam source, rot in hell"
45.3.200.10     IN A  127.0.0.3
                IN TXT "spam source, rot in hell"
; a complete range 10.222.5.0/24 using * wildcard
*.5.222.10      IN A  127.0.0.10
                IN TXT "confirmed DUL range, please use your ISP's smart mail host"
; a complete range 10.223.0.0/16 using * wildcard
*.223.10        IN A  127.0.0.10
                IN TXT "confirmed DUL range, please use your ISP's smart mail host"
;

Step 4: After you are done, reload the nameserver with
rndc reload example.com

or whatever your reload/restart procedure is.
Immediately check the log files for possible errors. As a quick check do a dig or nslookup locally on the server machine for “example.com” and “2.0.0.127.dnsbl.example.com”. Then do a dig or nslookup on a remote machine for “example.com” (should resolve, but most likely from a remote cache, so you cannot be sure at this time that you didn’t screw up anything) and “2.0.0.127.dnsbl.example.com” (should not resolve from remote!).
Then again check your new DNS setup for “example.com” with www.dnsreport.com to see if you have screwed up your basic DNS or not (don’t check “dnsbl.example.com” as this is running privately on one server only (the default would be 2 nameservers) and shouldn’t reply to public queries at all).

Step 5: Reconfigure your server’s mailserver to use the new local DNSBL you just created.

We are done.

This example may easily be adapted to host the DNSBL on a separate server.

Disclaimer: the above eventually worked for me. If you try to implement the above, you will do so at your own risk. Be careful not to screw up your server/s or parts of the internet’s DNS services. No guarantees, and yes, your mileage may vary.
If you have any doubts, don’t do it.

Building a Poor Man’s Barracuda – cPanel edition

This article will show you how to replicate most of the functionality of a Barracuda Spam Firewall on you cPanel server. The BSF has 12 layers that messages would be process through as illustrated in the graphic below. I’ll show you how to replicate most of these steps using customizations to the Exim and Spam Assassin. This guide assumes you have a basic working knowledge of cPanel, Linux, Exim, and Spam Assassin.

I’m going to outline my steps slightly differently than on the Barracuda so you get a better idea of what steps is configured where.

Connection Management:

  • Step 1 – Network Denial of Service Protection
  • Step 2 – Exim Rate Control
  • Step 3 – IP Reputation Analysis
  • Step 4 – Recipient Verification
  • Step 5 – Sender Verification

Content Scanning:

  • Step 6 – Content (Virus and Spam) Scanning
  • Step 7 – Fingerprint Analysis
  • Step 8 – Intent Analysis
  • Step 9 – Image Analysis
  • Step 10 – Bayesian Analysis
  • Step 11 – Rule-Based Scoring
  • Step 12 – Custom Policy

The text in bold type at the start of each step comes from the Barracuda Spam Firewall Email Filtering Whitepaper

Step 1: Network Denial of Service Protection

Built on a hardened and secure operating system, the Barracuda Spam Firewall receives email on behalf of the organization, insulating the organization’s email server from receiving direct Internet connections and the associated threats.

As filtering is being done on your cPanel server there is probably not going to be any front end device on your network to do DOS prevention. If not then I would recommend trying out CSF/CXS with blocklists enabled. You can also parse other blocklists from FireHOL and use ipset-rbl script. Deflate is no longer around anymore, will save that for another post.

Step 2: Rate Control

Automated spam software can be used to send large amounts of email to a single email server. To protect the email infrastructure from these flood-based attacks, the Barracuda Spam Firewall counts the number of incoming connections from a particular IP address and throttles the connections once a particular threshold is exceeded.

Organizations that relay email through known servers or communicate frequently with known partners can and should add the IP addresses of those trusted relays and good email servers to the Rate Control exemption list.

There’s several things we can do to control the rate at which Exim accepts connections. At Main >> Service Configuration >> Exim Configuration Editor you want to set:

Dictionary attack protection – On
Reject remote mail sent to the server’s hostname – On
Ratelimit suspicious SMTP servers – On
SpamAssassin™: ratelimit spam score threshold – <choose a score, I use 10. # This limits the rate a an IP can send mail at to 1/hr if the sent mail scores above the value set.
Log sender rates in the exim mainlog – On
Maximum message recipients (soft limit) # This isn’t set by default
Maximum message recipients before disconnect (hard limit) # This isn’t set by default

I also suggest putting in an SMTP banner delay to help stop ratware. At At Main >> Service Configuration >> Exim Configuration Manager, change “Introduce a delay into the SMTP transaction for unknown hosts and messages detected as spam.” to On.

 

Step 3: IP Reputation Analysis

  • Barracuda Reputation. Barracuda Reputation is maintained by Barracuda Central and includes a list of IP addresses of known, good senders as well as known spammers. Updates to the Barracuda IP Reputation database are delivered to the Barracuda Spam Firewall via Barracuda Energize Updates.
  • External block lists. The Barracuda Spam Firewall enables administrators to take advantage of external block lists which are also known as real-time block lists (RBLs) or DNS block lists (DNSBLs). Several organizations maintain external block lists, such as spamhaus.org.
  • Customer-defined policy for allowed IP addresses. The Barracuda Spam Firewall enables administrators to define a list of trusted email servers by IP address. By adding IP addresses to this list, administrators can avoid spam scanning of good email, both reducing processing requirements and eliminating the chances of false positives.
  • Customer-defined policy for blocked IP addresses. The Barracuda Spam Firewall also enables administrators to define a list of bad email senders. In some cases, administrators may choose to utilize the IP block lists to restrict specific email servers as a matter of policy rather than as a matter of spam protection.

I’m not sure what the BSF does for HELO checks, but I’m including HELO checks and DNS RBL (Real-time Block List) checks into one step.

There are several settings that now need to be enabled at Main >> Service Configuration >> Exim Configuration Manager:

Reject remote mail sent to the server’s hostname – On
Require HELO before MAIL – On
Require remote (hostname/IP) HELO – On
Require remote (domain) HELO – On
Require RFC-compliant HELO – On # Requires a HELO to be dotted like server.domain.com instead of just ‘server’

I don’t recommend enabling the options to reject SPF or DKIM failures at SMTP time. I feel they would lead to a lot of false positives. It’s better to ‘score’ these failures later with SpamAssassin instead.

Through trial and error, I’ve found the best RBLs to use are b.barracudacentral.org (BSF’s publicly available list), cbl.abuseat.org, and zen.spamhaus.org. All lists are publicly usable, however with SpamHaus if you run a commercial business you will need to pay. I recommend that if you have over 1000 user accounts, you should really go ahead and use rsync to copy down the lists so that way your MX server(s) are contacting your local RBL server instead of sending all those requests out to other servers on the internet. Setting up an RBL server is beyond the scope of this article, but you can use a program called rbldnsd to do it and can use instructions at the RBL providers to get started.

To enable RBLs in cPanel, to go to Service Configuration »Exim Configuration Manager, then set:

RBL: zen.spamhaus.org – On
(I do not recommend enabling SpamCop as it falses a lot)

Next click on “Manage Custom RBLs”. We now want to add a few custom RBLS. WHM asks for the RBL name, DNS list, and Info URL. On my server I use the following:

 

Manage-Custom-RBLs

Your mileage may vary, but those have worked well for me, especially having b.barracuda and zen.spamhaus. I wouldn’t recommend enabling SpamCop as I’ve seen a lot of false positives on it in the past. The other lists generally will blacklist an IP if it’s a known spam sender/bot, whereas SpamCop will block an legitimate business IP based on one user reporting spam from that sender (which may not actually be spam). In other words, there’s a much higher chance from spamhaus and barracuda that the sending IP is actually a spammer or infected machine.

To use the barracuda list you will need to register at http://www.barracudacentral.org/account/register
which will prevent your nameservers from possibly being blocked

*** IMPORTANT NOTE REGARDING DNS LOOKUPS ***

Some lists like spamhaus don’t return a result if the lookup comes from a public resolver.

Make sure your server is set to use either your datacenter’s resolvers, or you can configure named to accept recursive requests from 127.0.0.1, then you can use 127.0.0.1 as your resolver. If you use spamhaus and a public resolver then no mail will be rejected.

Step 3b: Greylisting

cPanel now supports doing greylisting. Greylisting will temporarily defer new SMTP connections from a sending IP. With greylisting, your server will defer the message with ‘due to greylisting’, then the sending server has to re-try the message. If the sending server sends the message again after your server’s configured retry time, then the message will come through. If it re-sends too fast then the message is deferred again. Greylisting can be configured in WHM at Home >> Email >> Greylisting.

When I enabled greylisting, my caught SPAM went from 25/day to about 3/day, so greylisting does really work to cut down to amount of SPAM that going to come in to your server.

While greylisting is enabled serverwide in WHM, it can be configured on a per cPanel account basis, and even a per domain basis in the cPanel account.

 

Step 4: Recipient Verification

Many spammers attack email infrastructures by harvesting email addresses. The Barracuda Spam Firewall verifies the validity of recipient email addresses through multiple techniques.

  • Protocol compliance. Similar to Sender Authentication, a recipient is first validated for being specified properly. An example of an enforcement policy includes, forcing RFC 821 compliance.
  • Custom policies. Organizations can define their policies based on allowed recipient email addresses for which spam scanning should be disabled. They can also define their own block lists based on email addresses. Note that allow lists override block lists.
  • LDAP recipient verification. Customers of Barracuda Spam Firewall models 300 and higher can choose to reject messages if the recipient email addresses do not appear in the LDAP directory.
  • SMTP recipient verification. By default, the Barracuda Spam Firewall rejects messages if the downstream mail server does not accept mail for that recipient.
  • Domain Keys. The Barracuda Spam Firewall enables administrators to inspect email messages for DomainKeys (DKIM) and take action when messages fail signature verifcation.

Now that the sender has given a proper HELO, they are not in a blacklist, now we need to see if the address they are sending to exists. So that you understand the importance of recipient verification, let me give you a quick example situation. Let’s say a spammer is blasting out garbage using the address john@doe.com. Without recipient verification, what will happen is that your server will accept the message regardless of whether the recipient exists and try to deliver it. If  “john@doe.com” is sending to address “jane@smith.com” and the jane address doesn’t exist, your server will then generate a mailerdaemon bounce and send it back to john@doe.com. Unfortunately for the REAL john@doe.com, he is going to get a lot of these type of non-existent user messages. Exim will hold the connection open from the sender and contact your mail host to see if jane@smith.com actually exists. If she does then the message will continue to be checked in Exim (we’re still only on step 4 now), but if jane@smith.com does not exist, Exim will spit back a Recipient Verify Failed message to the sending host therefore  the real John@doe.com never gets a ‘fake bounceback message’

cPanel servers can do recipient verification. The first thing to do is to go to Tweak Settings and check the value of “Initial default/catch-all forwarder destination”. It defaults to “System account” instead of “Fail”. Change this to Fail, that way mail to non-existent users will be rejected at SMTP time and recipient verification will work correctly. Do be aware though that cPanel accounts can change that option in their interface at cPanel >> Default Address.

 

Step 5: Sender Authentication

Declaring an invalid “from” address is a common practice by spammers. The Barracuda Spam
Firewall utilizes a number of techniques to both validate the sender as well as apply policy.

  • Protocol compliance. First and foremost, the sender is validated for being specified properly. Examples of enforcement policies include, forcing RFC 821 compliance or requiring fully-qualified domain names.
  • DNS lookup. To prevent senders from faking a “from” domain, a DNS lookup is performed on the sender domain to ensure that the domain exists.
  • Sender spoof protection. The Barracuda Spam Firewall has the option to prevent “spoofing” of an organization’s own domain by disallowing emails using that domain name to be sent from outside the organization. Note that sender spoof protection should not be enabled if the organization sends messages from outside their internal email infrastructure (e.g., in the case of marketing bulk-mail services).
  • Custom policies. Organizations can define their own allowed sender domains or email addresses. They can also define their own block lists based on sender domains or email addresses. Note that allow lists override block lists.
  • Sender policy framework (SPF). SPF is a proposed standard with growing momentum, designed to prevent spoofing of email domains. SPF provides a means for organizations to declare their known email servers in their DNS records so that email recipients can validate the identity of the sender domain based on the IP address of the sending email server. The Barracuda Spam Firewall enables email administrators to block or tag messages on failed SPF checks.

Yes I know, the BSF has Sender Authentication before Recipient Verification. I prefer doing a sender verify check AFTER the recipient verify check, because otherwise, you are checking to see if senders exist when they may be sending to non-existent users on your sysem (ie. dictionary attack). By reversing their order, we’re verifying that a recipient exists before we waste a call out to the sender’s MX to see if they exist. cPanel servers have two options here to consider, the first is “Sender Verification Callouts”, the second is “Sender Verification”. I used to recommend that Sender Verification Callouts be enabled so each sender would be validated, however that now will get your server blacklisted very quickly. I do however still recommend that “Sender Verification” be enabled as that option will just verify the existence of the sending domain without verifying the specific sending email address exists in their MX server.

 

Step 6: Virus Scanning

Virus Scanning takes precedence over all other Mail Scanning techniques and is applied even when mail passes through the Connection Management layers. As such, even email coming from “whitelisted” IP addresses, sender domains, sender email addresses or recipients are still scanned for viruses and blocked if a virus is detected.

This section will give you MIME checking, file extension blocking, virus scanning, and spam filtering. I used Clam Antivirus because it’s free and I’m familiar with it. You can certainly use whichever one you wish. To enable ClamAV in WHM, go to  Main >> cPanel >> Manage Plugins, then check the box “Install and keep updated ” next to ClamAV and click save, that will then install ClamAntivirus and configure Exim to use it.

The next thing is back at the Exim Configuration page, you want to enable the options to do attachment and mime scanning:

System Filter File – /etc/cpanel_exim_system_filter
Attachments: Filter messages with dangerous attachments – On

 

Step 7: Custom Policy (User Specified Rules)

Administrators can choose to define their own policies, perhaps for compliance or governance reasons, which take precedence over spam blocking rules delivered to the system automatically through Barracuda Energize Updates. The Barracuda Spam Firewall enables administrators to set custom content filters based on the subject, message headers, message bodies and attachment file type. In general, administrators do not need to set their own filters for the purposes of blocking spam, as these forms of rules are delivered to Barracuda Spam Firewalls automatically through Barracuda Energize Updates.

In this section I would recommend that you reject messages that score greater than 15. I’ve found 15 is a good value as I’ve not seen any legitimate mail score over 10. I recommend the following:

SpamAssassin™ reject spam score threshold – 15
Automatically whitelist known mobile device providers – On

There are several Access Lists that you can use to put IPs in to bypass certain checks:

Blacklisted SMTP IPs
# IPs from which SMTP connections are dropped unconditionally

Sender verification bypass IPs
# IPs for which to bypass SMTP-time sender verification checks

Only-verify-recipient SMTP hosts/IPs
# Hosts/IPs for which to bypass all SMTP-time checks except recipient verification

Trusted SMTP IPs
#IPs for which to bypass all SMTP-time recipient/sender/spam/relay checks

Backup MX hosts
# Hosts with reverse DNS from which connections are allowed regardless of rate limits.

 

Step 8: Fingerprint Analysis

A message “fingerprint” is based on commonly used message components (e.g., an image) across many instances of spam. Fingerprint analysis is often as a useful mechanism to block future instances of spam once an early outbreak is identified. Engineers at Barracuda Central work around the clock to identify new spam fingerprints which are then updated on all Barracuda Spam Firewalls through hourly Barracuda Energize Updates.

There’s several different plugins for Spam Assassin that check against a database of messages reported as spam. We’ll be setting up DCC, Pyzor, Razor, and iXhash. All of those plugins will need to be installed from source as they are not available in the CentOS / Red Hat repositories and cPanel doesn’t support third party repositories.

DCC -Distributed Checksum Clearinghouses

cd /usr/local/src
wget http://www.rhyolite.com/dcc/source/dcc.tar.Z
tar -zxvf dcc.tar.Z
cd dcc-1.3.141
./configure
make
make install

Uncomment “loadplugin Mail::SpamAssassin::Plugin::DCC” in /etc/mail/spamassassin/v310.pre

Add these 2 lines to /etc/mail/spamassassin/local.cf:

use_dcc 1
dcc_timeout 10

Pyzor – collaborative, networked system to detect and block spam using identifying digests of messages

yum install python-setuptools
cd /usr/local/src
(wget it from sourceforge.net)
tar -zxvf pyzor*.tar.gz
cd pyzor-0.5.0
python setup.py build
python setup.py install
/usr/bin/pyzor discover

You can verify pyzor is able to contact it’s server with the command “/usr/bin/pyzor ping” which should return “public.pyzor.org:24441    (200, ‘OK’)”

Uncomment “loadplugin Mail::SpamAssassin::Plugin::Pyzor” in /etc/mail/spamassassin/v310.pre
Add these 2 lines to /etc/mail/spamassassin/local.cf:

use_pyzor 1

Razor – distributed, collaborative, spam detection and filtering network

cd /usr/local/src
(wget it from sourceforge.net)
tar -jxvf razor-agents*.bz2
cd razor-agents-2.84
perl Makefile.PL
make
make test
make install

Uncomment “loadplugin Mail::SpamAssassin::Plugin::Razor2” in /etc/mail/spamassassin/v310.pre
Nothing needs to be added to local.cf for this.

iXhash – Similar in function to the above plugins

cd /usr/local/src
(wget it from sourceforge.net)
tar -zxvf  ihash*.tgz
cd iXhash-1.5.5
cp iXhash/iXhash.cf /etc/mail/spamassassin
cp iXhash/iXhash.pm /etc/mail/spamassassin
/usr/local/cpanel/3rdparty/bin/spamassassin -D IXHASH < iXhash.eml   # This will test to see if it works, look for IXHASH in the X-Spam-Status line

/scripts/restartsrv_spamd

 

Step 9: Intent Analysis

All spam messages have an “intent” – to get a user to reply to an email, visit a Web site or call a phone number. Intent analysis involves researching email addresses, Web links and phone numbers embedded in email messages to determine whether they are associated with legitimate entities. Frequently, Intent Analysis is the defense layer that catches phishing attacks. The Barracuda Spam Firewall features multiple forms of Intent Analysis.

  • Intent analysis. Markers of intent, such as URLs, are extracted and compared against a database maintained by Barracuda Central, and then delivered to the Barracuda Spam Firewall via hourly Barracuda Energize Updates.
  • Real-time intent analysis. For new domain names that may come into use, Real-Time Intent Analysis involves performing DNS lookups against known URL block lists.
  • Multilevel intent analysis. Use of free Web sites to redirect to known spammer Web sites is a growing practice used by spammers to hide or obfuscate their identity from mail scanning techniques such as Intent Analysis. Multilevel Intent Analysis involves inspecting the results of Web queries to URLs of well-known free Web sites for redirections to known spammer sites.

Intent analysis is done with the URIDNSBL plugin.  URIDNSBL looks up URLs found in the message against several DNS blocklists. cPanel already enables this option in /etc/mail/spamassassin/local.cf.

 

Step 10: Image Analysis

Today, image spam represents about one third of all traffic on the Internet. While Fingerprint Analysis captures a significant percentage of images after they have been seen, the Barracuda Spam Firewall also uses Image Analysis techniques which protect against new image variants. These techniques include:

  • Optical character recognition (OCR). Embedding text in images is a popular spamming practice to avoid text processing in anti-spam engines. OCR enables the Barracuda Spam Firewall to analyze the text rendered inside the images.
  • Image processing. To mitigate attempts by spammers to foil OCR through speckling, shading or color manipulation, the Barracuda Spam Firewall also utilizes a number of lightweight image processing technologies to normalize the images prior to the OCR phase. More heavyweight image processing algorithms are utilized at Barracuda Central to quicklygenerate fingerprints that can be used by Barracuda Spam Firewalls to block messages.
  • Animated GIF analysis. In addition, the Barracuda Spam Firewall contains specialized algorithms for analyzing animated GIFs for suspect content.

There are several plugins to do this, I used to use fuzzy_ocr, but it’s no longer maintained. Other plugins that looks to be active haven’t been updated in several years either.

 

Step 11: Bayesian Analysis

Bayesian Analysis is a linguistic algorithm that profiles language used in both spam messages and legitimate email for any particular user or organization. To determine the likelihood that a new email is spam, Bayesian Analysis compares the words and phrases used in the new email against the corpus of previously identified email.

With cPanel 11.36 paths to certain files have changed, so we need to create a symlink:
cd /usr/local/bin
ln -s /usr/local/cpanel/3rdparty/bin/sa-learn .

I createf a script you run to train the spam filter, updating the bayesian token database. You can put it in /etc/cron.daily/ . For your IMAP accounts, if you put SPAM mail that isn’t caught and tagged with ***SPAM*** in a folder called ‘spam’ and mail that is tagged but shouldn’t be in ‘ham’, the script will then use those messages to train the filter so it performs better. If the user has an existing prefs file then you’ll need to add the contents of the script generated file to it, otherwise you can just copy it to /home/user/.spamassassin/user_prefs, but don’t forget to change ownership of that file!

In each user’s user_prefs file, you may want to adjust the scores for BAYES_95 and BAYES_99 so that mail that has such a large percentage chance of being SPAM is scored higher:

score BAYES_95 5
score BAYES_99 7

 

Step 12: Rule-based Scoring

Beyond absolute blocks that a single filter can apply, the Barracuda Spam Firewall also includes a sophisticated scoring engine that weighs multiple factors where a single filter may result into restrictive policy. By combining multiple rules with known weightings, the Barracuda Spam Firewall can deliver a strong confidence interval for spam messages. The Barracuda Spam Firewall enables administrators to set global spam scores. Certain models of the Barracuda Spam Firewall also support per domain and per user thresholds.

Spam Assassin custom plugins and tests can be enabled by editing /etc/mail/spamassassin/local.cf. You can also set custom scores in the same file. For cPanel users themselves, they can adjust scores by adding the proper line to /home/user/.spamassassin/user_prefs, but they cannot enable a plugin that way or add custom tests.

 

 

Final thoughts and steps

Building a Poor Man’s Barracuda is the first step building a successful anti-spam solution. Next on the list are:

  1. Bayes Training with sa-learn cron script based in each user’s Inbox, Sent and Spam folders.
  2. Custom rules with MailSpike BL and WL, Scoring based on lists like SORBS, URIBL, XBL, CBL, Spamhaus, Spamcop etc (Scoring NOT in exim level denying)
  3. Enabling Spamhaus DQS
  4. Custom DNSBL. Creating your custom blacklist (bl.isp.com) and your custom Whitelist (wl.isp.com). Score based too. We don’t want to deny everything. Just to end in “spam” folder.

 

systemd. wait. what?

Systemd is a cancer and it will keep spreading until the host is dead. Just like cancer it’s not malicious, it sincerely believes it knows better than the healthy cells it’s infecting, but that doesn’t make it less lethal.