[clamav-users] ign2 whitelist don't work

None of those examples are signatures, they are engine driven detections. You must disable Heuristics using clamd.conf and clamscan options.

Sent from Janet's iPad

-Al-

Post by Reindl Harald
Hi
* the follwoing rules don't make anything but troubles
* created a ign2 file
* again a reject of clamav-milter
* tried also whitelist "Eicar-Test-Signature"
* also still hits
why?!
_______________________________________________________
Heuristics.Phishing.Email.SpoofedDomain
Heuristics.Email.SSL-Spoof
Phishing.Heuristics.Email.SpoofedDomain
Phishing.Heuristics.Email.SSL-Spoof
Heuristics.Encrypted.PDF
_______________________________________________________
Fri Jul 15 16:42:46 2016 -> fd[10]: Heuristics.Phishing.Email.SpoofedDomain(007f163a4f71a336e78174b48e14bc0a:10951) FOUND

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Reindl Harald

2016-07-16 11:40:13 UTC

Post by Al Varnell
None of those examples are signatures, they are engine driven detections

not entirely true - here are running two instances of clamd, one with
3rd party rules scored in spamassassin and the other one with the
official sigfiles - only the one with the official hits

sadly that also means move them to the scoring instance would leave not
much rules / signatures for the milter as last ressort

Post by Al Varnell
You must disable Heuristics using clamd.conf and clamscan options.

that's not a useful answer since the only option is
"HeuristicScanPrecedence" which don't disable anything and so "you must
do this" without saying how is pointless

"PhishingScanURLs no" would also disable "safebrowsing.cvd" and likely
also most of the 3rd party rules

disable heuristics entirely (given there would be an an option) would
also disable "Heuristics.OLE2.ContainsMacros"

it makes no sense that you can't disable specific heuristics
_______________________________________________________

such false positives are *unacceptable* in case of the monthly account
overview and frankly i have not seen any hit which was not very likely a
false positive (as example newsletters from payment companies over
services like mailchimp)

Jul 8 14:42:49 mail-gw spamd[16295]: spamd: result: . -3 -
BAYES_50,CUST_DNSWL_5_ORG_N,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HTML_IMAGE_RATIO_06,HTML_MESSAGE,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_PASS,USER_IN_DEF_DKIM_WL

Jul 8 14:42:10 mail-gw postfix/cleanup[19493]: 3rmDds0LjczB44:
milter-reject: END-OF-MESSAGE from
mta106b.pmx1.epsl1.com[142.54.244.106]: 5.7.1 Virus found or dangerous
attachment: "Heuristics.Phishing.Email.SpoofedDomain";
from=<bounce-***@mail.paypal.at>
to=<*****> proto=ESMTP helo=<mta106b.pmx1.epsl1.com>

Jul 8 14:42:49 mail-gw postfix/cleanup[19119]: 3rmDfY2gcSzB44:
milter-reject: END-OF-MESSAGE from
mta103b.pmx1.epsl1.com[142.54.244.103]: 5.7.1 Virus found or dangerous
attachment: "Heuristics.Phishing.Email.SpoofedDomain";
from=<bounce-***@mail.paypal.at>
to=<****> proto=ESMTP helo=<mta103b.pmx1.epsl1.com>

Post by Al Varnell

Charles Swiger

2016-07-18 16:14:11 UTC

Post by Al Varnell
You must disable Heuristics using clamd.conf and clamscan options.

For that specific case, check that OLE2BlockMacros is set to no.

it makes no sense that you can't disable specific heuristics

This is a reasonable point. One should be able to use the ign2 whitelist to disable specific heuristics, as well as having more fine-grained control within clamd.conf.

In the meantime, however, if you want to have PhishingScanURLs or PUA categories enabled, then you should use any matches returned under Heuristics.* or Phishing.Heuristics.* as a soft fail and use it for scoring purposes rather than as a hard fail.

_______________________________________________________
such false positives are *unacceptable* in case of the monthly account overview and frankly i have not seen any hit which was not very likely a false positive (as example newsletters from payment companies over services like mailchimp)

[ ... ]
While I'd agree that it should be unacceptable for financial institutions to use third parties for email distribution of sensitive content, I suspect that wasn't your intended point.

I also believe that it should not be acceptable for financial institutions-- or anyone who values their reputation-- to send emails containing HTML links where the domain of the link text shown to the user does not match the domain of the actual A href attribute.

Have you contacted PalPal and asked them to investigate why they are sending emails which look like phishing attempts via epsl1.com domain, which domain not only doesn't appear in the SPF records for paypal.com / paypal.at, but also doesn't appear to have any published MX records at all (per "dig -t mx epsl1.com.")...?

I did. And I have since stopped using PayPal entirely after they failed to improve their practices. The more people who contact them via <***@paypal.com> both for phishing attempts and for supposedly genuine mail from PayPal which triggers AV phishing warnings, the better.

Regards,

--
-Chuck

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Reindl Harald

2016-07-18 17:03:58 UTC

Post by Al Varnell
You must disable Heuristics using clamd.conf and clamscan options.

For that specific case, check that OLE2BlockMacros is set to no.

the point is this should be independent

it makes no sense that you can't disable specific heuristics

This is a reasonable point. One should be able to use the ign2 whitelist to disable specific heuristics, as well as having more fine-grained control within clamd.conf.

fine-grained control would be difficult given you have a mix of 3rd
party signatures to know which affects what

i am really shocked that ign2 whitelists don't work in a simple way that
whatever is triggered due the scan is compared agianst the whitelist and
just skipped as it was never there

Post by Charles Swiger
In the meantime, however, if you want to have PhishingScanURLs or PUA categories enabled, then you should use any matches returned under Heuristics.* or Phishing.Heuristics.* as a soft fail and use it for scoring purposes rather than as a hard fail.

running as milter that's not possible and when you run clamd only as
spamassassin plugin any whitelisting would skip the complete virus scan
what is the reason to have the milter as second instance with different
signatures

the current beahvior is completly unflexible even with having both clamd
as spamassasin-plugin and one as "last resort" milter

[ ... ]
While I'd agree that it should be unacceptable for financial institutions to use third parties for email distribution of sensitive content, I suspect that wasn't your intended point.

they use SPF and until clamav can't do a SPF test it must not run
phising tests unconditionally while the real problem is that
"PhishingScanURLs no" also disabled google safe browing signatures

Post by Charles Swiger
I also believe that it should not be acceptable for financial institutions-- or anyone who values their reputation-- to send emails containing HTML links where the domain of the link text shown to the user does not match the domain of the actual A href attribute.

frankly i am sorry that i can't play police for every institution out
there and explain them how to handle email :-)

Post by Charles Swiger
Have you contacted PalPal and asked them to investigate why they are sending emails which look like phishing attempts via epsl1.com domain

since i am mailadmin and don't have access to my users email no
however - "epsl1.com" is *not* the sending domain
you confuse hostnames and sender address

Post by Charles Swiger
which domain not only doesn't appear in the SPF records for paypal.com / paypal.at, but also doesn't appear to have any published MX records at all (per "dig -t mx epsl1.com.")...?

sorry but you don't understand SPF really good and since when have MX
records something to do with outbound mail

mail.paypal.at. 3600 IN TXT "v=spf1
ip4:142.54.244.96/27 ip4:142.54.244.128/29 -all"

142.54.244.106 is the sending IP
again: i am a mailadmin and can't babysit every communication between
customers and their communication channels, i can only look at the SPF
which was correct and so say it is a false-positive and in that case the
fault of clamav

Charles Swiger

2016-07-19 14:02:16 UTC

Post by Charles Swiger
For that specific case, check that OLE2BlockMacros is set to no.

the point is this should be independent

Well, it currently isn't.

Post by Reindl Harald
it makes no sense that you can't disable specific heuristics

This is a reasonable point. One should be able to use the ign2 whitelist to disable specific heuristics, as well as having more fine-grained control within clamd.conf.

fine-grained control would be difficult given you have a mix of 3rd party signatures to know which affects what

That's why being able to list the exact signatures to ignore rather than having fixed categories in clamd.conf would be a better option.

Post by Reindl Harald
i am really shocked that ign2 whitelists don't work in a simple way that whatever is triggered due the scan is compared agianst the whitelist and just skipped as it was never there

Yes.

The milter approach is less flexible. With a scoring mechanism, you can rate actual viruses sufficiently negative that the scoring algorithm will always reject them.

Post by Reindl Harald
such false positives are *unacceptable* in case of the monthly account overview and frankly i have not seen any hit which was not very likely a false positive (as example newsletters from payment companies over services like mailchimp)

[ ... ]
While I'd agree that it should be unacceptable for financial institutions to use third parties for email distribution of sensitive content, I suspect that wasn't your intended point.

they use SPF and until clamav can't do a SPF test it must not run phising tests unconditionally while the real problem is that "PhishingScanURLs no" also disabled google safe browing signatures

ClamAV is a virus scanner; it is not its job to perform SPF lookups. Whatever is calling ClamAV to process mail, whether it be a milter interface, amavisd, etc is the software that should be handling SPF checks.

frankly i am sorry that i can't play police for every institution out there and explain them how to handle email :-)

You have no general obligation to police the entire internet, but one should care about the institutions that you or your userbase interacts with.

Post by Charles Swiger
Have you contacted PalPal and asked them to investigate why they are sending emails which look like phishing attempts via epsl1.com domain

since i am mailadmin and don't have access to my users email no
however - "epsl1.com" is *not* the sending domain
you confuse hostnames and sender address

Perhaps English isn't your native language?

I spoke precisely; I did not say that epsl1.com was the sending domain. Your logs demonstrate that it, or more precisely mta106b.pmx1.epsl1.com was the MTA sending the message to your mailserver and that mail.paypal.at was the SMTP "Mail from" domain.

Two points:

1) I got different SPF results here, although hitting a third-party website to lookup the SPF records for mail.paypal.at does give the result you've shown. (I'm travelling and using someone else's network at the moment, so it's possible that they/their ISP is swizzling DNS lookups.)

2) In the absence of MX records stating otherwise, I expect that any mailserver which sends outbound email should be willing to accept inbound mail for the same domains it terminates or relays email on behalf of.

Post by Reindl Harald
again: i am a mailadmin and can't babysit every communication between customers and their communication channels, i can only look at the SPF which was correct and so say it is a false-positive and in that case the fault of clamav

Well, I've been a mail admin myself since 1990 or so. Someone with an interest in Usenet archeology could probably find on the order of 7000 posts to comp.mail.sendmail by me.

ClamAV rejected the message with Heuristics.Phishing.Email.SpoofedDomain; one can run clamscan --debug against the mail (assuming you have a copy kept around in quarantine) to see exactly what is happening.

It is extremely likely that the email contained misleading HTML links, presumably intended as a click-through for tracking / marketing campaign purposes. For an example, see:

http://lists.clamav.net/pipermail/clamav-users/2015-August/001857.html

Post by Reindl Harald
LibClamAV debug: Looking up hash
5E5978396FC0F81B1032CDA256B95D0D65EA0605DBE0643E89231C049A337640 for
urldefense.
proofpoint.com/(26)v2/url?u=http-3A__www.bankofamerica.com_emaildisclaimer&d=AwMFAg&c=ewHkv9vLloTwhsKn5d4bTdoqsmB
fyfooQX5O7EQLv5TtBZ1CwcvjU063xndfqI8U&r=2aYd0Z__pii05laLdA-SVeMDDGgKztEldmYeWZkrEInUKhhOQFnXGHbtYgd15gmS&m=1gyane
8UIsmcsdK0OgwckCpz8Guf1pgeNHHmOLXQn5Y&s=XYG3vPf_ZUZQe7myUa6pQ8SUpYmn9GNeGK33YzupujA&e=(293)
https://urldefense.proofpoint.com->http://www.bankofamerica.com

Those sort of links are dangerous because phishing attempts try to do exactly the same sort of thing. SPF isn't relevant to ClamAV, since AFAIK ClamAV doesn't perform SPF lookups at all.

Regards,

--
-Chuck

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Reindl Harald

2016-07-19 14:28:23 UTC

Post by Charles Swiger
Perhaps English isn't your native language?

no, it isn't

first: i know that SPF is not relevant for clamav, but since it's a
clean way to verify the source of a message and clamav can't do this
such spoofing rules in clamav which can't be disabled without disable
other things too is bad

second: i know about scoring - as said i have more than one clamd and
what i had to do now is move safebrowsing rules to the other instance
because they whould have disabled too by "PhishingScanURLs no"

yes, i know that the milter is not flexible and *hence* there are
different clamd instances with different configs and different
signatures - *but* the last ressort instance for clamav-milter is
terrible unflexible to configure because the weakness of ign2 which is
waht this topic is about and hence i would nearly need 3 instances to
get the granualry i need - but that's not really doable because
currently the two running instances eare eating around 800 MB RAM

Post by Charles Swiger
I spoke precisely; I did not say that epsl1.com was the sending domain. Your logs demonstrate that it, or more precisely mta106b.pmx1.epsl1.com was the MTA sending the message to your mailserver and that mail.paypal.at was the SMTP "Mail from" domain.

1) I got different SPF results here, although hitting a third-party website to lookup the SPF records for mail.paypal.at does give the result you've shown. (I'm travelling and using someone else's network at the moment, so it's possible that they/their ISP is swizzling DNS lookups.)

or you did "dig TXT paypal.at" which is not the same as "dig TXT
mail.paypal.at"

anyways - "dig NS paypal.at"
paypal.at. 300 IN NS ns2.p57.dynect.net.
paypal.at. 300 IN NS ns1.p57.dynect.net.
paypal.at. 300 IN NS ns3.p57.dynect.net.
paypal.at. 300 IN NS ns4.p57.dynect.net.

dig TXT mail.paypal.at @ns2.p57.dynect.net
mail.paypal.at. 3600 IN TXT "v=spf1
ip4:142.54.244.96/27 ip4:142.54.244.128/29 -all"

Post by Charles Swiger
2) In the absence of MX records stating otherwise, I expect that any mailserver which sends outbound email should be willing to accept inbound mail for the same domains it terminates or relays email on behalf of.

that is not how email works

a) the sender is @mail.paypal.at and not "@epsl1.com"
b) every smarter setup these days has strictly
seperated outbound and inbound servers

what you expect is completly pointless - as example you have no business
to deliver mail to our outbound server unless you are a customer with a
valid username and password since inbound mail is expected at the MX
(spamfirewall) and not at the submission server

why?

because it's much easier to define MTA policies for spamfiltering when
you need not to mix with mail clients and when you do outbound
spamfiltering you need completly different rules (no RBL looksups, no
PTR checks, different scorings and first of all no postscreen in front
which a MUA can't handle)

Kris Deugau

2016-07-19 14:39:45 UTC

Post by Charles Swiger
The milter approach is less flexible. With a scoring mechanism, you can rate actual viruses sufficiently negative that the scoring algorithm will always reject them.

That depends on the milter you're using. My own favoured milter is
MIMEDefang, which allows you do do anything you like to a message in
transit so long as you can figure out how to code it in Perl.

ClamAV hits on any of the Heuristics.* tests get flagged instead of
treated the same as the signature-based hits, and that flag either
causes an an adjustment in the SpamAssassin results returned directly to
MIMEDefang later on, or a header is added which I check for in
SpamAssassin on mail delivery.

-kgd
_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Charles Swiger

2016-07-19 17:00:07 UTC

On Jul 19, 2016, at 10:28 AM, Reindl Harald <***@thelounge.net> wrote:
[ ... ]

that is not how email works

As I recall, you were either submitting a bug report about ClamAV and SPF, which seems misguided as you've since acknowledged ("i know that SPF is not relevant for clamav"), or at the least you were looking for feedback about how to better handle legitimate email from paypal.at which you were bouncing due to ClamAV's heuristics.
True.

Post by Reindl Harald
b) every smarter setup these days has strictly
seperated outbound and inbound servers

False. Assuming that there is only one correct mail architecture is a major fallacy.

What you describe is one reasonable architecture for a large ISP which needs to have redundant sending and receiving mail servers. However, there are lots of smaller sites which have no need for that-- they might be better off having an external MX relay in their firewall DMZ which handles both inbound and outbound mail, and an internal mailhost / reader box, for example.

Post by Reindl Harald
what you expect is completly pointless - as example you have no business to deliver mail to our outbound server unless you are a customer with a valid username and password since inbound mail is expected at the MX (spamfirewall) and not at the submission server

You appear to have skipped past this phrase: "In the absence of MX records stating otherwise..."

If a mail server sends outbound, it needs to be willing to handle bounces and DSNs for those messages/domains which it sends.

Post by Reindl Harald
why?
because it's much easier to define MTA policies for spamfiltering when you need not to mix with mail clients and when you do outbound spamfiltering you need completly different rules (no RBL looksups, no PTR checks, different scorings and first of all no postscreen in front which a MUA can't handle)

It is reasonable to have different inbound and outbound MTAs to implement different policies? Sure.

Is that the only mechanism by which one can have different policies? Nope.

It is reasonable to trust all local mail and push the burden of checking it upon others? Nope.

You should be applying spamfiltering and especially malware/virus scanning to outbound email just as rigorously as you do to inbound email. In a few cases that I am familiar with, outbound email is screened more carefully than inbound email.

Regards,

--
-Chuck

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Reindl Harald

2016-07-19 17:09:43 UTC

Post by Charles Swiger
[ ... ]

that is not how email works

no, i was submitting what the subject says and explained why it's
unacceptable not to be able in a software which tries to make
assumptions about phising by no clue about SPF

Post by Charles Swiger
True.

Post by Reindl Harald
b) every smarter setup these days has strictly
seperated outbound and inbound servers

False. Assuming that there is only one correct mail architecture is a major fallacy.

bla - yes there are more ways but your whole stuff about SPF was
entirely wrong from the very begin in case of the messages in question

Post by Charles Swiger
If a mail server sends outbound, it needs to be willing to handle bounces and DSNs for those messages/domains which it sends.

bullshit - the MX does and this servers outbound mail was *not* for a
domain below it's own hostname and so it has no business for inbound mail

It is reasonable to have different inbound and outbound MTAs to implement different policies? Sure.
Is that the only mechanism by which one can have different policies? Nope.

far off-topic the whole discussion just because you where unable to look
careful at the one logline and make correct SPF requests while i already
told in the orginal mail that i have verified it and even posted the
spamassassin SPF_PASS line of the message in question

Post by Charles Swiger
It is reasonable to trust all local mail and push the burden of checking it upon others? Nope.

did i say that?

Post by Charles Swiger
You should be applying spamfiltering and especially malware/virus scanning to outbound email just as rigorously as you do to inbound email. In a few cases that I am familiar with, outbound email is screened more carefully than inbound email.

where did i say anything else?

but you need different configs as i explained and it should be pretty
clear why - there is no point makeing dialup-rbl-tests on a submission
client which is typically a enduser somewhere at home

Charles Swiger

2016-07-19 17:15:01 UTC

Post by Kris Deugau

Post by Charles Swiger
The milter approach is less flexible. With a scoring mechanism, you can rate actual viruses sufficiently negative that the scoring algorithm will always reject them.

That depends on the milter you're using. My own favoured milter is
MIMEDefang, which allows you do do anything you like to a message in
transit so long as you can figure out how to code it in Perl.

Fair enough. clamav-milter was implied by context, but MIMEDefang is
certainly a decent choice, especially if one can do a bit of Perl hacking.

For what it is worth, I've been happier with Python-based filters, since
they tend to use less resources than the Perl-based software. But that
has more to do with how well people can write code in the different
languages-- I've also seen some very lightweight and efficient Perl code.

Post by Kris Deugau
ClamAV hits on any of the Heuristics.* tests get flagged instead of
treated the same as the signature-based hits, and that flag either
causes an an adjustment in the SpamAssassin results returned directly to
MIMEDefang later on, or a header is added which I check for in
SpamAssassin on mail delivery.

Are you using LMTP, or did SpamAssassin grow a local delivery agent capability?

Last I'd seen, amavisd-new or MailScanner are still recommended for integration
between the MTA and SpamAssassin / ClamAV....

Regards,

--
-Chuck

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Charles Swiger

2016-07-19 17:19:45 UTC

Post by Charles Swiger
False. Assuming that there is only one correct mail architecture is a major fallacy.

bla - yes there are more ways but your whole stuff about SPF was entirely wrong from the very begin in case of the messages in question

You managed to misinterpet what I actually said, and are now off in the weeds.

Have fun with that strawman.

Post by Charles Swiger
If a mail server sends outbound, it needs to be willing to handle bounces and DSNs for those messages/domains which it sends.

bullshit - the MX does and this servers outbound mail was *not* for a domain below it's own hostname and so it has no business for inbound mail

That, and you are inexcusably rude to people who were trying to help you.

Good day, sir. I hope your customers find better service elsewhere.

Regards,

--
-Chuck
_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Reindl Harald

2016-07-19 17:27:11 UTC

Post by Charles Swiger
False. Assuming that there is only one correct mail architecture is a major fallacy.

bla - yes there are more ways but your whole stuff about SPF was entirely wrong from the very begin in case of the messages in question

You managed to misinterpet what I actually said, and are now off in the weeds.
Have fun with that strawman.

you just said nothing but nonsense in context of that messages and
trigger a "heuristic phising" in case of a paypal mail from a mailserver
listed in the SPF of the paypal spf belonging to the envelope sender is
a false postive

not be able to whitelist such a misbehaving trigger without disable
other things too is a bug and/or design mistake - period

Post by Charles Swiger
If a mail server sends outbound, it needs to be willing to handle bounces and DSNs for those messages/domains which it sends.

bullshit - the MX does and this servers outbound mail was *not* for a domain below it's own hostname and so it has no business for inbound mail

That, and you are inexcusably rude to people who were trying to help you.

trying to help?
where?

well, agreed, more useful than the first response "You must disable
Heuristics using clamd.conf " while such a otpion don't exist

you are talking 90% of this thread completly off-topic stuff

Post by Charles Swiger
Good day, sir. I hope your customers find better service elsewhere

i doubt - the problem itself is solved for days but in a way which
should not be needed and the whole conversation with you was to 90% or
more completly off-topic

Kris Deugau

2016-07-19 18:05:39 UTC