2017-04-08 - Re: [GRASE-Hotspot] Semi-Urgent - coova-chilli[1382]: net.c: 780: 11 (Resource temporarily unavailable) could not read packet

Header Data

From: Michael Raynor <mx***1@gmail.com>
Message Hash: 0e3640d77a80a8d4f14e215e2b89e69336edae55b76b09919f2b3196587ad78d
Message ID: <dbafe41d-3766-4b36-b6ea-a8bfe774bd11@grasehotspot.org>
Reply To: <CAESLx0+b=g6C7JMQbdvpbgbTZd62Ur=sBB48SPPFQSmYdjWcog@mail.gmail.com>
UTC Datetime: 2017-04-08 05:18:32 UTC
Raw Date: Sat, 08 Apr 2017 05:18:32 -0700

Raw message 

Hi Tim,

Thanks for the reply.

Details are in https://paste.grasehotspot.org/view/4ed0b655

I did reboot the server to add additional compute and memoy earlier in the 
day but kep /var/log/syslog pointing to /dev/null (as a precaution.

I have removed the symlink and have fixed the permissions and those log 
entries are no longer showing.

It seems to "go crazy" when there are a huge number of 
connections/concurrent connections - the peak busy time.

I have also added the follow line to /etc/sysctl.conf to see if it was an 
issue with empemeral ports (however there was already ~32K outbound ports 
in there before the change
net.ipv4.ip_local_port_range = 5000 65535

Let me know if you need further details. I will leave the logging as normal 
overnight.

Cheers,

Michael

On Saturday, 8 April 2017 20:24:47 UTC+10, timwhite88 wrote:
>
> Does a reboot fix it?
>
> What version of Coova-chilli is that running? (
> https://grasehotspot.org/support/collecting-support-information/)
>
> Regards
>
> On Sat, Apr 8, 2017 at 7:34 AM, Michael Raynor <mx***.@gmail.com 
> <javascript:>> wrote:
>
>> Hi Tim,
>>
>> We have four big conferences happening back to back over the next four 
>> weeks.
>>
>> I received an alert from New Relic telling me that the grase servers' 
>> disk was full.
>>
>> Having a look I saw that /var/log/syslog was 79GB and was full of the 
>> following messages - 
>>
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 780: 11 (Resource 
>> temporarily unavailable) could not read packet
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 806: 11 (Resource 
>> temporarily unavailable) net_read_eth(fd=5, len=65497, mtu=1500) == -1
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 780: 11 (Resource 
>> temporarily unavailable) could not read packet
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 806: 11 (Resource 
>> temporarily unavailable) net_read_eth(fd=5, len=65497, mtu=1500) == -1
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 780: 11 (Resource 
>> temporarily unavailable) could not read packet
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 806: 11 (Resource 
>> temporarily unavailable) net_read_eth(fd=5, len=65497, mtu=1500) == -1
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 780: 11 (Resource 
>> temporarily unavailable) could not read packet
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 806: 11 (Resource 
>> temporarily unavailable) net_read_eth(fd=5, len=65497, mtu=1500) == -1
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 780: 11 (Resource 
>> temporarily unavailable) could not read packet
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 806: 11 (Resource 
>> temporarily unavailable) net_read_eth(fd=5, len=65497, mtu=1500) == -1
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 780: 11 (Resource 
>> temporarily unavailable) could not read packet
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 806: 11 (Resource 
>> temporarily unavailable) net_read_eth(fd=5, len=65497, mtu=1500) == -1
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 780: 11 (Resource 
>> temporarily unavailable) could not read packet
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 806: 11 (Resource 
>> temporarily unavailable) net_read_eth(fd=5, len=65497, mtu=1500) == -1
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 780: 11 (Resource 
>> temporarily unavailable) could not read packet
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 806: 11 (Resource 
>> temporarily unavailable) net_read_eth(fd=5, len=65497, mtu=1500) == -1
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 780: 11 (Resource 
>> temporarily unavailable) could not read packet
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 806: 11 (Resource 
>> temporarily unavailable) net_read_eth(fd=5, len=65497, mtu=1500) == -1
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 780: 11 (Resource 
>> temporarily unavailable) could not read packet
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 806: 11 (Resource 
>> temporarily unavailable) net_read_eth(fd=5, len=65497, mtu=1500) == -1
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 780: 11 (Resource 
>> temporarily unavailable) could not read packet
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 806: 11 (Resource 
>> temporarily unavailable) net_read_eth(fd=5, len=65497, mtu=1500) == -1
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 780: 11 (Resource 
>> temporarily unavailable) could not read packet
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 806: 11 (Resource 
>> temporarily unavailable) net_read_eth(fd=5, len=65497, mtu=1500) == -1
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 780: 11 (Resource 
>> temporarily unavailable) could not read packet
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 806: 11 (Resource 
>> temporarily unavailable) net_read_eth(fd=5, len=65497, mtu=1500) == -1
>> Apr  8 01:51:48 grase01 coova-chilli[1382]: net.c: 780: 11 (Resource 
>> temporarily unavailable) could not read packet
>>
>> The syslog file is still growing as I write this and I am afraid it will 
>> run out again very soon.
>>
>> Thanks for your help!
>>
>> Michael
>>
>> -- 
>> This mailing list is for the Grase Hotspot Project 
>> http://grasehotspot.org
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Grase Hotspot" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to gr***.@grasehotspot.org <javascript:>.
>> To post to this group, send email to gr***.@grasehotspot.org 
>> <javascript:>.
>> Visit this group at 
>> https://groups.google.com/a/grasehotspot.org/group/grase-hotspot/.
>> To view this discussion on the web visit 
>> https://groups.google.com/a/grasehotspot.org/d/msgid/grase-hotspot/acff143d-0233-476e-a254-e795559735b2%40grasehotspot.org 
>> <https://groups.google.com/a/grasehotspot.org/d/msgid/grase-hotspot/acff143d-0233-476e-a254-e795559735b2%40grasehotspot.org?utm_medium=email&utm_source=footer>
>> .
>>
>
>

Thread