[clamav-users] clamd server '/var/run/clamd.amavisd/clamd.sock' gave '' response

Discussion:

(too old to reply)

Alex

2016-02-21 23:40:43 UTC

Hi,

I have a clamav-0.99-2 installation on fedora23 and periodically I
receive a message when running clamav-notify-servers after having run
freshclam that reports:

# clamav-notify-servers
clamd server '/var/run/clamd.amavisd/clamd.sock' gave '' response

I have a script that periodically rsyncs the malwarepatrol db to the
/var/lib/clamav directory then runs the clamav-notify-servers. I
believe the problem is related to this occurring at the same time as
the regular freshclam-sleep script running clamav-notify-servers.

Is this the intended behavior for clamd?

I have about 9M signatures now, so it appears to take a long time to
reload the database every time the clamav-notify-servers signal is
sent.

Can someone provide some advice on the best way to do this? I don't
think I can control the timing of the clamav-notify-servers to make
sure it doesn't happen while another instance occurs. Should I just
redirect the output to /dev/null?

Is it common to have 9M entries?

It looks to take about 30s to reload the database:
Feb 21 03:22:15 mail03 clamd[1006]: Reading databases from /var/lib/clamav
Feb 21 03:22:46 mail03 clamd[1006]: Database correctly reloaded
(8888331 signatures)
Feb 21 03:22:46 mail03 clamd[1006]: Client disconnected (FD 23)

This is on a six-core 3Ghz system on SSD disks.

[***@mail03 clamav]# ls
badmacro.ndb foxhole_filename.cdb phishtank.ndb
spamattach.hdb
blurl.ndb foxhole_generic.cdb porcupine.hsb
spamimg.hdb
bofhland_cracked_URL.ndb hackingteam.hsb porcupine.ndb
spam.ldb
bofhland_malware_attach.hdb javascript.ndb rogue.hdb
spearl.ndb
bofhland_malware_URL.ndb junk.ndb safebrowsing.cvd
spear.ndb
bofhland_phishing_URL.ndb jurlbla.ndb sanesecurity.ftm
winnow.attachments.hdb
my_sigwhitelist.gdb jurlbl.ndb scamnailer.ndb
winnow_bad_cw.hdb
my_sigwhitelist.ign2 lott.ndb scam.ndb
winnow.complex.patterns.ldb
my_sigwhitelist.wdb main.cvd
securiteinfoascii.hdb winnow_extended_malware.hdb
bytecode.cld malwarehash.hsb securiteinfo.hdb
winnow_malware.hdb
crdfam.clamav.hdb malwarepatrol.ndb
securiteinfohtml.hdb winnow_malware_links.ndb
create_sig.txt mirrors.dat securiteinfo.ign2
winnow_phish_complete_url.ndb
daily.cld phish.ndb sigwhitelist.ign2
winnow_spam_complete.ndb

I think the commercial securiteinfo databases are entirely too large
and don't perform very well.

Of course I could cut down on the databases, but I'm more interested
in finding out why clamd produces the error message when multiple
signals are sent.

Thanks,
Alex
_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Alex

2016-02-22 14:08:43 UTC

Permalink

Hi,

Can’t be of much help with your primary issue, but to answer one or your questions, the official ClamAV database is a bit over 4 million. I can’t conceive of a situation where you would need every conceivable unofficial database, but then I have no idea what you are doing with your setup, other than it would appear to have some relationship to e-mail service.

It comes from complaints from users about zero-day and cryptowall
viruses making it through the mail gateway, then being caught by
Symantec as it reaches Exchange. Or a compromise being traced back to
not having caught a virus a few hours earlier.

There was a discussion less than a month ago concerning minimum essential database subscriptions, so
suggest you search around in the archive for that thread
<clamav-user archives>.

I'll search around, thanks.

Assistance with my other issues would still very much be appreciated.

Thanks,
Alex

-Al-

Post by Alex
Hi,
I have a clamav-0.99-2 installation on fedora23 and periodically I
receive a message when running clamav-notify-servers after having run
# clamav-notify-servers
clamd server '/var/run/clamd.amavisd/clamd.sock' gave '' response
I have a script that periodically rsyncs the malwarepatrol db to the
/var/lib/clamav directory then runs the clamav-notify-servers. I
believe the problem is related to this occurring at the same time as
the regular freshclam-sleep script running clamav-notify-servers.
Is this the intended behavior for clamd?
I have about 9M signatures now, so it appears to take a long time to
reload the database every time the clamav-notify-servers signal is
sent.
Can someone provide some advice on the best way to do this? I don't
think I can control the timing of the clamav-notify-servers to make
sure it doesn't happen while another instance occurs. Should I just
redirect the output to /dev/null?
Is it common to have 9M entries?
Feb 21 03:22:15 mail03 clamd[1006]: Reading databases from /var/lib/clamav
Feb 21 03:22:46 mail03 clamd[1006]: Database correctly reloaded
(8888331 signatures)
Feb 21 03:22:46 mail03 clamd[1006]: Client disconnected (FD 23)
This is on a six-core 3Ghz system on SSD disks.
badmacro.ndb foxhole_filename.cdb phishtank.ndb
spamattach.hdb
blurl.ndb foxhole_generic.cdb porcupine.hsb
spamimg.hdb
bofhland_cracked_URL.ndb hackingteam.hsb porcupine.ndb
spam.ldb
bofhland_malware_attach.hdb javascript.ndb rogue.hdb
spearl.ndb
bofhland_malware_URL.ndb junk.ndb safebrowsing.cvd
spear.ndb
bofhland_phishing_URL.ndb jurlbla.ndb sanesecurity.ftm
winnow.attachments.hdb
my_sigwhitelist.gdb jurlbl.ndb scamnailer.ndb
winnow_bad_cw.hdb
my_sigwhitelist.ign2 lott.ndb scam.ndb
winnow.complex.patterns.ldb
my_sigwhitelist.wdb main.cvd
securiteinfoascii.hdb winnow_extended_malware.hdb
bytecode.cld malwarehash.hsb securiteinfo.hdb
winnow_malware.hdb
crdfam.clamav.hdb malwarepatrol.ndb
securiteinfohtml.hdb winnow_malware_links.ndb
create_sig.txt mirrors.dat securiteinfo.ign2
winnow_phish_complete_url.ndb
daily.cld phish.ndb sigwhitelist.ign2
winnow_spam_complete.ndb
I think the commercial securiteinfo databases are entirely too large
and don't perform very well.
Of course I could cut down on the databases, but I'm more interested
in finding out why clamd produces the error message when multiple
signals are sent.
Thanks,
Alex

_______________________________________________
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadm

Groach

2016-02-22 14:34:52 UTC

Permalink

FWIW, if I may offer opinion: I would agree with Alex with the need to
source out better unofficial databases (such as sanesecurity,
securiteinfo etc): clam definitions are inherently slow on the uptake
of new threats, taking a day or more (at best) and in some cases never
(ive demonstrated this in other posts on this mail list). (I dont know
if I am breaking a rule here by advertising - apologies if so but I am
not affiliated in any way) sanesecurity definitions have proven
invaluable and EXTREMELY responsive regarding zero-hour threats
especially to the crytolocker and DOC-macro based threats (REALLY being
'zero hour' coverage by definition). (In my opinion. I also believe
that they the ONLY way you can use ClamAv is if you employ 3rd party
definitions given that most threats are at its highest danger level just
hours after release and its simply unproductive releasing signatures to
catch them days later (or never!). Clam always needs supplementing with
either 3rd party definitions (which Alex has), another realtime scanning
commercial Av product (which Alex has), or both (which Alex has).

I can bery well imagine, though that 9 million definitions are excessive
and probably over 70% have no point in existing any more (threat
probably been and gone).

Alex, I use standard defs and sane only (to cut down on definitions) and
have excellent coverage without the slow 'startup times' that you have
mentioned whilst being backed up with Bitdefender on the client PC's.
Might be worth changing your signatures if they are increasing your
start time without adding any noticeable benefit (or at least
experiemnting to see if it makes a difference. You can always revert
back to your current choice....and your " ' ' response " error. ;-)

Jim

Post by Alex
Hi,

There was a discussion less than a month ago concerning minimum essential database subscriptions, so
suggest you search around in the archive for that thread
<clamav-user archives>.

I'll search around, thanks.
Assistance with my other issues would still very much be appreciated.
Thanks,
Alex

-Al-

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

Dennis Peterson

2016-02-22 16:30:29 UTC

Permalink

# grep FOUND /var/log/clamav/clamd.log* |grep -c UNOFFICIAL
80
# grep FOUND /var/log/clamav/clamd.log* |grep -v -c UNOFFICIAL
0
# grep FOUND /var/log/clamav/clamd.log* |grep -c -i sanesecurity
38
# grep FOUND /var/log/clamav/clamd.log* |grep -c -i winnow
42

My logs go back only to January, but this is a typical pattern for the last 7
years or so. Notice that official sigs have not found anything. Important too to
know that because of cpu cost scanning is the last thing done to test mail and
that most rejections happen prior and scanning isn't performed. In terms of
effectiveness, proactive prevention using hosts.deny, iptables, sendmail access,
j-chkmail milter (includes regex, urlbl, heuristics, spam traps), IP reputation,
and reactive denial with deny-hosts utility, fail2ban, manual scanning of log
reports.

I've not looked at the code to see if ClamAV has a signature order (theirs first
then "unofficial") but it is certainly possible that if Sane Security signatures
were not installed that ClamAV signatures may get more hits.

dp

FWIW, if I may offer opinion: I would agree with Alex with the need to source

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Groach

2016-02-22 17:06:31 UTC

Permalink

I dont think there is any 'cause' to be had (that the unofficial
signatures found threats and that the official ones didnt) other than
ClamAV signatures are too few, too ineffective and more importantly too
late.

I ran AV for 3 years as an inline mail scanner and it didnt catch a
single threat in my emails. Not one SINGLE one. In 3 years! (Although
there were WAY too many false positives when scanning my hard drives
(almost daily.) In November, after some testing, I decided on
implementing and using Sane signatures and the difference was immediate
within the FIRST HOUR of turning them on. Now we must have on average
at least 5 of 6 emails DAILY with threats attached to them and they get
caught immediately by the unofficial signatures. The daily threat of
'bad-macro' in Office documents (cryptolocking) was caught at retrieval
and never got through to the users (thereby removing the risk of them
stupidly opening it, enabling macros in Office, and wondering how pretty
that red "you have been encrypted, send us your money" screen looks).
These emails were always coming in almost daily before implementing Sane
but ClamAV definitions just didnt have any clue (or urgency!) on dealing
with them. In 3 months only 2 email threats managed to come in just
before my hourly definition update and therefore got through.

So I have no doubt, that even if ClamAV definitions took priority in the
database, it wouldnt have mattered as they had the efficacy of wearing
sandals for rain boots.

Post by Dennis Peterson
# grep FOUND /var/log/clamav/clamd.log* |grep -c UNOFFICIAL
80
# grep FOUND /var/log/clamav/clamd.log* |grep -v -c UNOFFICIAL
0
# grep FOUND /var/log/clamav/clamd.log* |grep -c -i sanesecurity
38
# grep FOUND /var/log/clamav/clamd.log* |grep -c -i winnow
42
My logs go back only to January, but this is a typical pattern for the
last 7 years or so. Notice that official sigs have not found anything.
Important too to know that because of cpu cost scanning is the last
thing done to test mail and that most rejections happen prior and
scanning isn't performed. In terms of effectiveness, proactive
prevention using hosts.deny, iptables, sendmail access, j-chkmail
milter (includes regex, urlbl, heuristics, spam traps), IP reputation,
and reactive denial with deny-hosts utility, fail2ban, manual scanning
of log reports.
I've not looked at the code to see if ClamAV has a signature order
(theirs first then "unofficial") but it is certainly possible that if
Sane Security signatures were not installed that ClamAV signatures may
get more hits.
dp

Post by Groach
FWIW, if I may offer opinion: I would agree with Alex with the need
to source out better unofficial databases (such as sanesecurity,

_______________________________________________
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Joel Esler (jesler)

2016-02-22 18:57:02 UTC

Permalink

Gentlemen. We get the point. We’re working on it. I had a conversation with the malware lead last week to see what we can do here.
--
Joel Esler
Manager, Talos Group

On Feb 22, 2016, at 12:06 PM, Groach <groachmail-***@yahoo.com<mailto:groachmail-***@yahoo.com>> wrote:

I dont think there is any 'cause' to be had (that the unofficial signatures found threats and that the official ones didnt) other than ClamAV signatures are too few, too ineffective and more importantly too late.

I ran AV for 3 years as an inline mail scanner and it didnt catch a single threat in my emails. Not one SINGLE one. In 3 years! (Although there were WAY too many false positives when scanning my hard drives (almost daily.) In November, after some testing, I decided on implementing and using Sane signatures and the difference was immediate within the FIRST HOUR of turning them on. Now we must have on average at least 5 of 6 emails DAILY with threats attached to them and they get caught immediately by the unofficial signatures. The daily threat of 'bad-macro' in Office documents (cryptolocking) was caught at retrieval and never got through to the users (thereby removing the risk of them stupidly opening it, enabling macros in Office, and wondering how pretty that red "you have been encrypted, send us your money" screen looks). These emails were always coming in almost daily before implementing Sane but ClamAV definitions just didnt have any clue (or urgency!) on dealing with them. In 3 months only 2 email threats managed to come in just before my hourly definition update and therefore got through.

So I have no doubt, that even if ClamAV definitions took priority in the database, it wouldnt have mattered as they had the efficacy of wearing sandals for rain boots.

On 22/02/2016 17:30, Dennis Peterson wrote:
# grep FOUND /var/log/clamav/clamd.log* |grep -c UNOFFICIAL
80
# grep FOUND /var/log/clamav/clamd.log* |grep -v -c UNOFFICIAL
0
# grep FOUND /var/log/clamav/clamd.log* |grep -c -i sanesecurity
38
# grep FOUND /var/log/clamav/clamd.log* |grep -c -i winnow
42

My logs go back only to January, but this is a typical pattern for the last 7 years or so. Notice that official sigs have not found anything. Important too to know that because of cpu cost scanning is the last thing done to test mail and that most rejections happen prior and scanning isn't performed. In terms of effectiveness, proactive prevention using hosts.deny, iptables, sendmail access, j-chkmail milter (includes regex, urlbl, heuristics, spam traps), IP reputation, and reactive denial with deny-hosts utility, fail2ban, manual scanning of log reports.

I've not looked at the code to see if ClamAV has a signature order (theirs first then "unofficial") but it is certainly possible that if Sane Security signatures were not installed that ClamAV signatures may get more hits.

dp

On 2/22/16 6:34 AM, Groach wrote:
FWIW, if I may offer opinion: I would agree with Alex with the need to source out better unofficial databases (such as sanesecurity, securiteinfo etc):

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/v

Alex

2016-02-22 20:48:53 UTC

Permalink

Hi,

Post by Joel Esler (jesler)
Gentlemen. We get the point. We’re working on it. I had a conversation with the malware lead
last week to see what we can do here.

Can you help with my original question about:

clamd server '/var/run/clamd.amavisd/clamd.sock' gave '' response

Is this expected, when clamav-notify-servers is run while clamd is
re-reading the databases?

Can someone tell me if this happens on their system too? It was a bug
a long time ago, but thought it was fixed. It's just started again.

Post by Joel Esler (jesler)
I dont think there is any 'cause' to be had (that the unofficial signatures found threats and that the official ones didnt) other than ClamAV signatures are too few, too ineffective and more importantly too late.

I never saw this message. Was this posted to the list?

I've found the sanesecurity rules to work well. The securiteinfo rules
are horrible. I'd never expect to only use the default clamav rules.

Thanks,
Alex
_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav

Groach

2016-02-22 21:24:07 UTC

Permalink

Yes. Youll get the jist of it in this post:
http://lists.clamav.net/pipermail/clamav-users/2016-February/002351.html
(I say they are 3x false positives but I got in a tiz and meant to say
3x Malwares (reported by me) - look for the 3 links to virustotal in the
post)

Post by Alex
Hi,

I never saw this message. Was this posted to the list?
Thanks,
Alex

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml

Dennis Peterson

2016-05-26 12:33:43 UTC

Permalink

Forgot to respond to this earlier - this can happen if an update begins before a
previous update finishes. And this can happen if you have multiple scripts
fetching signatures from multiple vendors. Some scripts have a built in random
delay that attempts to prevent every user from updating on the hour, for
example, but because these scripts are not aware of each other the opportunity
exists that there will be overlap.

The tools communicate with the clamd daemon via a unix or tcp socket and sent
the word "PING". The daemon should respond with "PONG" and it will unless it is
busy. That can cause the connection to timeout without a response and that
produces the error message you see.

There is probably a more elegant way of handling this short of a serializing
layer, but since the purpose of the script is to request a reload and not be
part of a service monitoring tool, I think the correct response is to give up
quietly and not obsess. Delaying until the previous reload is completed then
launching another reload simply extends the time the service is unavailable.

Summary: For the rare occasion that multiple vendors have new signatures
available at the same time, the possibility exists that the fetching processes
will result in an overlap of reload requests. The clamd daemon becomes
disfunctional during a reload so it is in everyone's interest to minimize the
number of this these are called. This suggests the reload request should be
isolated from the fetch process so that excessive reloads are not requested - a
simple serializing process can manage this to avoid a self-induced DOS.
Especially problematic on Solaris SPARC systems and systems with memory/cpu
limitations.

dp

Post by Alex
Hi,

Post by Joel Esler (jesler)
Gentlemen. We get the point. We’re working on it. I had a conversation with the malware lead
last week to see what we can do here.

clamd server '/var/run/clamd.amavisd/clamd.sock' gave '' response
Is this expected, when clamav-notify-servers is run while clamd is
re-reading the databases?
Can someone tell me if this happens on their system too? It was a bug
a long time ago, but thought it was fixed. It's just started again.

I never saw this message. Was this posted to the list?
I've found the sanesecurity rules to work well. The securiteinfo rules
are horrible. I'd never expect to only use the default clamav rules.
Thanks,
Alex
_______________________________________________
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml

_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav

Dennis Peterson

2016-05-27 19:45:03 UTC

Permalink

In addition to what has been discussed, the selfcheck that clamd does can
overlap a freshclam or other signature process and produce the same warning.
This is particularly true for signature installers or admins that don't do
atomic file operations. That is to say, if you scp/sftp/mv/copy files into the
clamav signature folder, a non-atomic operation, clamd has a good chance of
noticing them as they arrive and will perform a self-reload. As I recall the
selfcheck default is every 600 seconds so that provides a lot of opportunity to
create reload interference.

dp
_______________________________________________
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq

http://www.clamav.net/contact.html#ml