[clamav-users] Strange problem with custom Yara rule

[kionez]

Hi,

I'm going mad with a strange behaviour of clamav with custom yara rules.

I'm trying to match some nasty spam email, I decided to use yara for my
custom rules but i noticed a problem: if I use only string detect clamav
(either via clamscan or clamdscan) matches all the email (text +
headers) but if I use regex detect it only matches email's text.

For example:

$mail_header = /X-Mailer: PHPMailer 5\.2\./

doesn't match, but:

$mail_header = "X-Mailer: PHPMailer 5.2."

matches.. I tryed to "reduce" the match to only "ailer", but the
situation doesn't change, even appending a "nocase" flag.

Am I wrong or there's something strange? :)


k.
_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Steven Morgan]

Hi,

The first question is: Do you have pcre installed and was it found by
ClamAV .\configure? You should see something like:

pcre : /usr

near the end of the ./configure output.

Steve
_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[kionez]

#include <Steven Morgan.h> // created 13/04/2016 16:37

Post by Steven Morgan
Hi,
The first question is: Do you have pcre installed and was it found by
ClamAV .\configure?

[cut]

Ops, I forgot to mention my system configuration.. sorry.

I'm using it on my antispam server with Debian Jessie (with clamav
0.99+dfsg-0+deb8u2 and libpcre3 8.35-3.3+deb8u4 ) and also testing on my
laptop with Arch linux (clamav 0.99.1-2 and pcre 8.38-3). I try to
recompile clamav on my laptop and it founds pcre in /usr, as expected.

I think that pcre works fine, because I can match patterns on email's
content, but not in email headers (i.e.: X-Mailer:, From: etc)

I try to explain with an example, on http://pastebin.com/gixrDAXq you
should find the 3 file used (a simple email and two custom ruleset)

using rules from Yara:

$ yara -s test_header.yara test.eml
Header_string test.eml
0x1f1:$match: pass
0x2a2:$match: pass
Header_Regex test.eml
0x1f1:$match: pass
0x2a2:$match: pass

$ yara -s test_body.yara test.eml
Body_string test.eml
0x440:$match: test mailing
Body_Regex test.eml
0x440:$match: test mailing

using rules from clamav:

$ clamscan --no-summary -d test_header.yara -z test.eml
test.eml: YARA.Header_string.UNOFFICIAL FOUND

$ clamscan --no-summary -d test_body.yara -z test.eml
test.eml: YARA.Body_string.UNOFFICIAL FOUND
test.eml: YARA.Body_Regex.UNOFFICIAL FOUND

I think that test_header.yara have to match the "pass" string in header
part with that simple regex. I'm trying to debug it but my code
knowledge is limited, it seems that every time a regex is used the
header part is not parsed by clamav (also if I use a "string" rule with
a "regex" rule, ie: $match_string and $match_regex)


k.


_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Charles Swiger]

Hi, kionez--

Post by kionez
I'm using it on my antispam server with Debian Jessie (with clamav
0.99+dfsg-0+deb8u2 and libpcre3 8.35-3.3+deb8u4 ) and also testing on my
laptop with Arch linux (clamav 0.99.1-2 and pcre 8.38-3). I try to
recompile clamav on my laptop and it founds pcre in /usr, as expected.
I think that pcre works fine, because I can match patterns on email's
content, but not in email headers (i.e.: X-Mailer:, From: etc)

This doesn't mention how your email gets processed from your MTA into clamav.

Normally, things like amavisd extract the body of the message and any MIME
attachments, and scan those components only. They don't scan the entire message
or the mail headers.

See bypass_decode_parts and keep_decoded_original_maps for amavisd-new, or
look for the equivalent in whatever you are using to pass mail into the
virus scanning.

Regards,

[Steven Morgan]

Hi,

Thanks for the example. I've opened bug
https://bugzilla.clamav.net/show_bug.cgi?id=11552 to track.

Thanks again,
Steve
_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Kevin Lin]

ClamAV, in order to optimize the AC algorithm execution, runs the
filetype signatures alongside the malware detection signatures. ClamAV
is set to immediately return after AC execution if a filetype
signature detection occurs. This unfortunately causes the engine to
skip PCRE signature execution.

Post by Steven Morgan
Hi,
Thanks for the example. I've opened bug
https://bugzilla.clamav.net/show_bug.cgi?id=11552 to track.
Thanks again,
Steve
_______________________________________________
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[Kevin Lin]

Please refer to the bug report at:
https://bugzilla.clamav.net/show_bug.cgi?id=11552
for the patch to resolve the issue.

ClamAV, in order to optimize the AC algorithm execution, runs the filetype signatures alongside the malware detection signatures. ClamAV is set to immediately return after AC execution if a filetype signature detection occurs. This unfortunately causes the engine to skip PCRE signature execution.

Post by Steven Morgan
Hi,
Thanks for the example. I've opened bug
https://bugzilla.clamav.net/show_bug.cgi?id=11552 to track.
Thanks again,
Steve
_______________________________________________
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

[kionez]

#include <Kevin Lin.h> // created 13/04/2016 19:33

Post by Steven Morgan
https://bugzilla.clamav.net/show_bug.cgi?id=11552
for the patch to resolve the issue.

Wow, thanks for the quick solution :)

I've just tried the patch on my laptop and seems working fine, I do some
tests and I will push on production server (I think is pretty safe to
use it on production).

Some example, using files posted before and a simple rule for combining
string and regex match:

rule Header_combined // test_combined.yara
{
strings:
$match_string = "pass"
$match_regex = /pass/
condition:
$match_regex and $match_string
}

Before patch:

$ clamscan --no-summary -d test_body.yara -z test.eml
test.eml: YARA.Body_string.UNOFFICIAL FOUND
test.eml: YARA.Body_Regex.UNOFFICIAL FOUND

$ clamscan --no-summary -d test_header.yara -z test.eml
test.eml: YARA.Header_string.UNOFFICIAL FOUND

$ clamscan --no-summary -d test_combined.yara -z test.eml
test.eml: OK


After patch:

$ clamscan --no-summary -d test_body.yara -z test.eml
test.eml: YARA.Body_string.UNOFFICIAL FOUND
test.eml: YARA.Body_Regex.UNOFFICIAL FOUND

$ clamscan --no-summary -d test_header.yara -z test.eml
test.eml: YARA.Header_string.UNOFFICIAL FOUND
test.eml: YARA.Header_Regex.UNOFFICIAL FOUND

$ clamscan --no-summary -d test_combined.yara -z test.eml
test.eml: YARA.Header_combined.UNOFFICIAL FOUND

So, I confirm that is solved!

Thanks!

k.
_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml