ViRus

back to http://scratchpad.wikia.com/wiki/Sasecurity

I used to have the same problem, It is caused by the clients. If you have a client that has a poorly maintained computer i.e. no anti adware software or viruses then the computer is constantly sending spurious traffic to the internet. The nodes can not handle this and locks up – Splash restarting. A quick solve is to reboot the box. I found out which client was responsible and he did not have any anti adware or virus software. I installed ad-aware from lavasoft and an antivirus. We found that he had over 500 critical objects detected on ad-aware and over 30 viruses. This solved the problem. I am running the pro version which has a hardened splash to deal with this problem. I would suggest upgrading to the latest build and checking your clients computers for malicious software/adware/viruses.

This communication contains information, which is confidential and may also be privileged. It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. If you have received this communication in error please return it to the sender.

Every once in a while, or some times quite often, a node will stop passing any client traffic and will not accept wireless client associations. If you hook up a monitor directly to the node while this problem is happening, you'll see the message "error- restarting splash" displayed repeatedly. This condition persists until you reboot the box. There is more than one cause:

(1) There is a script in the /hj directory that's invoked by a cronjob every 10 minutes. This script checks to see of splashd is running by trying to curl the index page and look for html. We've found that sometimes curl return a blank page even if splashd is running properly, and it will kill and restart splashd unnecessarily. To fix this problem make the following change:

after the curl command, replace

if ["$a" = ""] then etc.

with

if ["$?" = "7"] then etc.

so that unless curl can't connect at all it won't invoke the splash restarter.

Qorvus has added this change to 2.6 (beta)

(2) In addition, Our engineers have found that this condition is sometimes related to a logic bug in SED, introduced by one of the system scripts. When this bug is triggered, the nocat.conf file in usr/local/etc will be empty. Nocat (splashd) needs this file to work properly and when it vanishes splashd fails and a seperate watchdog script will attempt to restart it every few minutes, without success (leading to the error message you see on the monitor).

asdfasdf
The trojan that was located on the clients PC's (two different client's at the same time) was an IRC back door (Trojan Backdoor.SdBot.27) which allows someone else to gain control of the PC via an IRC session that is built into the trojan. It did not carry enough bandwidth to hurt the system, but it did create enough network traffic to slow me down. Both clients have been "cleansed" and returned to service. Thanks to everyone that offered help! I can only hope that someday I can return the favor. And of course a special thanks to Kenny for all his time helping me and Larry for his help as well.

I wanted to take the opportunity to thank everyone that responded to my post (both on the list and off the list) earlier this week about the possible DOS on the Linden network. Poor Kyle has been working himself to death tracking down everything, and today, it looks as though the mesh is back to 100%. I suspected a possible DOS early on - but really did not know how to properly diagnose it.

found to be causing the problems was that two clients had backdoor trojans installed on their PC's. One was definitely conducting a DOS, while the other one was quietly uploading the contents of that PC's hard drive to someone at Cambridge university (based off the ip's that the traffic was going to). The client conducting a DOS was causing most of the trouble as the CPU load on that MeshBox (a heavily used junction on the network) would average 25%. All of the other units run an average of 3%. The DOS doesn't seem to eat up all of the bandwidth, but instead was overtaxing the CPU of the MeshBox, thereby slowing it to a crawl. Ping times would average 3 to 5 seconds instead of 30 milliseconds!

One interesting thing to point out was that the computer that was quietly uploading all of it's contents to Cambridge University was a new computer that the local computer sales store had set up. According to the proprietor, that computer was online for 10 minutes (unprotected) before he installed some sort of protection, and by that time someone had already exploited that computer and it had a virus. This client was using AVG and it reported the virus but would not remove it. The person thought nothing of it for some reason. Still, this highlights how vulnerable an unprotected PC can be.

How we did it: with iptables. Someone (forgive me - I forget) posted a message about 2 months ago about l33thax0rs on their network and how they removed them. I used the instructions from that message, and Kyle used Alec's suggestion to boot the offenders off the network.

tcpdump was also used quite extensively to gather information about what was going on. Kyle is using LinkFerret for monitoring traffic on the mesh and between him using that and me using tcpdump, we were able to spot the offensive traffic quite easily. Most nodes are going to generate a lot of traffic in tcpdump, which you can log to a file. The node where the DOS was occurring was generating unreal amounts of traffic. Like I said, it wasn't consuming all of the bandwidth... just overtaxing the CPU.

It's almost unbelievable that one or two clients can do this to a network, but under the circumstances, it makes sense. I wanted to point this out because it's likely that this can and does happen on other mesh networks.. so hopefully it will save someone some headaches.