DDoS attack


status

At 20th of November at 21:26 UTC my server suffered from a DDoS attack. The information I got from my ISP were :

	
Direction IN
Internal 5.9.158.75
Threshold Packets 100.000 packets/s
Sum 34.067.000 packets/300s (113.556 packets/s), 34.056 flows/300s (113 flows/s), 1,271 GByte/300s (34 MBit/s)
...
External (snipped ipv4), 2.000 packets/300s (3 packets/s), 1 flows/300s (0 flows/s), 0,000 GByte/300s (0 MBit/s)
External (snipped ipv4), 2.000 packets/300s (3 packets/s), 1 flows/300s (0 flows/s), 0,000 GByte/300s (0 MBit/s)
External (snipped ipv4), 2.000 packets/300s (3 packets/s), 1 flows/300s (0 flows/s), 0,000 GByte/300s (0 MBit/s)
...  
  
I couldn't connect to the system any longer. I didn't get any further information form my AS about the attack. Therefore at 22th of November I initiated an automatic server reboot at 09:57 UTC which succeeded w/o any problems.

From the kern.log:

	
Nov 20 22:26:29 tor-relay kernel: [2431358.124515] TCP: request_sock_TCP: Possible SYN flooding on port 80. Sending cookies.  Check SNMP counters.
Nov 20 22:26:48 tor-relay kernel: [2431377.216133] ------------[ cut here ]------------
Nov 20 22:26:48 tor-relay kernel: [2431377.216141] WARNING: CPU: 7 PID: 12421 at net/sched/sch_generic.c:303 dev_watchdog+0x272/0x280()
Nov 20 22:26:48 tor-relay kernel: [2431377.216143] NETDEV WATCHDOG: enp3s0 (r8169): transmit queue 0 timed out
Nov 20 22:26:48 tor-relay kernel: [2431377.216145] Modules linked in:
Nov 20 22:26:48 tor-relay kernel: [2431377.216148]  af_packet nf_log_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables nf_log_ipv4 nf_log_common xt_LOG xt_multiport nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables i2c_i801 i2c_core tpm_tis tpm thermal processor battery atkbd x86_pkg_temp_thermal button microcode fan
Nov 20 22:26:48 tor-relay kernel: [2431377.216173] CPU: 7 PID: 12421 Comm: emerge Not tainted 4.1.7-hardened-r1 #1
Nov 20 22:26:48 tor-relay kernel: [2431377.216174] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 0922 09/10/2012
Nov 20 22:26:48 tor-relay kernel: [2431377.216176]  ffffffff994fa966 0000000000000000 ffffffff99bced09 ffff88041fbc3d18
Nov 20 22:26:48 tor-relay kernel: [2431377.216179]  ffffffff99983e26 0000000000000000 ffff88041fbc3d68 ffff88041fbc3d58
Nov 20 22:26:48 tor-relay kernel: [2431377.216182]  ffffffff9947f08a ffff88041fbc3d48 ffffffff99bced09 000000000000012f
Nov 20 22:26:48 tor-relay kernel: [2431377.216185] Call Trace:
Nov 20 22:26:48 tor-relay kernel: [2431377.216187]    [] ? print_modules+0x76/0xe0
Nov 20 22:26:48 tor-relay kernel: [2431377.216198]  [] dump_stack+0x45/0x5d
Nov 20 22:26:48 tor-relay kernel: [2431377.216203]  [] warn_slowpath_common+0x8a/0xd0
Nov 20 22:26:48 tor-relay kernel: [2431377.216205]  [] warn_slowpath_fmt+0x5a/0x70
Nov 20 22:26:48 tor-relay kernel: [2431377.216210]  [] ? task_tick_fair+0x2a8/0x760
Nov 20 22:26:48 tor-relay kernel: [2431377.216213]  [] dev_watchdog+0x272/0x280
Nov 20 22:26:48 tor-relay kernel: [2431377.216216]  [] ? dev_deactivate_queue+0x70/0x70
Nov 20 22:26:48 tor-relay kernel: [2431377.216219]  [] call_timer_fn+0x47/0x140
Nov 20 22:26:48 tor-relay kernel: [2431377.216222]  [] run_timer_softirq+0x291/0x450
Nov 20 22:26:48 tor-relay kernel: [2431377.216224]  [] ? dev_deactivate_queue+0x70/0x70
Nov 20 22:26:48 tor-relay kernel: [2431377.216228]  [] __do_softirq+0xf8/0x290
Nov 20 22:26:48 tor-relay kernel: [2431377.216230]  [] irq_exit+0x9d/0xb0
Nov 20 22:26:48 tor-relay kernel: [2431377.216235]  [] smp_apic_timer_interrupt+0x55/0x70
Nov 20 22:26:48 tor-relay kernel: [2431377.216237]  [] apic_timer_interrupt+0x97/0xa0
Nov 20 22:26:48 tor-relay kernel: [2431377.216239]  
Nov 20 22:26:48 tor-relay kernel: [2431377.216241] ---[ end trace 93431a9382c0a11a ]---
Nov 20 22:26:48 tor-relay kernel: [2431377.237826] r8169 0000:03:00.0 enp3s0: link up
Nov 20 22:28:18 tor-relay kernel: [2431467.175659] r8169 0000:03:00.0 enp3s0: link up
Nov 20 22:28:30 tor-relay kernel: [2431479.172562] r8169 0000:03:00.0 enp3s0: link up
Nov 20 22:28:42 tor-relay kernel: [2431491.164472] r8169 0000:03:00.0 enp3s0: link up
Nov 20 22:28:54 tor-relay kernel: [2431503.170416] r8169 0000:03:00.0 enp3s0: link up
Nov 20 22:29:06 tor-relay kernel: [2431515.148333] r8169 0000:03:00.0 enp3s0: link up
Nov 20 22:29:18 tor-relay kernel: [2431527.143293] r8169 0000:03:00.0 enp3s0: link up
Nov 20 22:29:30 tor-relay kernel: [2431539.142164] r8169 0000:03:00.0 enp3s0: link up
Nov 20 22:29:42 tor-relay kernel: [2431551.124104] r8169 0000:03:00.0 enp3s0: link up
...
Nov 22 10:56:24 tor-relay kernel: [2562675.624512] r8169 0000:03:00.0 enp3s0: link up
  
The last line repeated and the network was down till I initiated a hardware reset.

?

The given numbers yields to 30,000 - 40,000 attacking systems each sending 3 packets/sec. FWIW a much more harder attack at 27th of April this year did not harm the system at all:

	
  Sum 104.275.000 packets/300s (347.583 packets/s), 1.441 flows/300s (4 flows/s), 11,177 GByte/300s (305 MBit/s)
  
It looks for me that the curent attack turned the network card into a state from which it couldn't recovered itself. Just a driver problem in the Linux kernel 4.1.7 ? Here's a link to the LKML with the corresponding thread.

Another fallout is, that syslog-ng v3.7.1 produces broken time stamps here - again ! That issue was expected to be fixed already with v3.6.4 (Gentoo bug 533328) :-/


back to my home page