Who's infected?

What's one sign that a machine is infected with some kind of malware? How about a quick test to see which machines are ARPing the most?

Spyware/worms/virii often try to connect to randomly selected machines on your network. When a machine tries to talk to a local IP address for the first time it sends an ARP packet to find out its ethernet ("MAC") address. On the other hand, normal (uninfected) machines generally only talk to a few machines: the servers they use and their local router. Detecting a machine that is sending considerably more ARP packets than other machines on the network is often a good sign that the machine is infected.

Let's build a simple shell pipeline to collect the next 100 ARP packets and determines which hosts generated them. Sort of a "most likely to ARP" award. I did this today and found 2 machines, one with 64 spyware "products". Shesh.

These commands should work on any Unix/Linux or Unix-like system. You will need the tcpdump command and root access. The command "which tcpdump" will tell you if you have tcpdump installed.

Here is the command I used:

tcpdump -l -n arp | egrep 'arp who-has' | head -100 | awk '{ print $NF }' |sort | uniq -c | sort -n

The output looked like this:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en0, link-type EN10MB (Ethernet), capture size 96 bytes
   1 192.168.1.104
   2 192.168.1.231
   5 192.168.1.251
   7 192.168.1.11
   7 192.168.1.148
   7 192.168.1.230
   8 192.168.1.254
  11 192.168.1.56
  21 192.168.1.91
  30 192.168.1.111
101 packets captured
3079 packets received by filter
0 packets dropped by kernel

Ignoring the headers, the middle lines show a count followed by an IP address. During our experiment host 192.168.1.111 sent 30 ARP packets, while 192.168.104 only sent 1. Most machines rarely ARPed in that time period but 2 hosts had 4-6 times as many ARPs as some of the other machines. Those were my two problem children. I quick run of some anti-malware software and they were good as new.

Here's how I built this command line.

I started with this command:

tcpdump -l -n arp

"tcpdump" listens to the local ethernet. The -l flag is required if we're going to pipe the output to another program because, unlike other programs, tcpdump does something special with buffering so that it runs faster. However, when piping the output we need it to act more normal. The -n means don't do DNS lookups for each IP address we see. The "arp" means that we only want it to display ARP packet.

Run he command yourself. In fact, you will learn more if you try each command as you read the article. Nothing here deletes any data. Of course, it may be illegal to snoop packets on your network, so be warned. Only do this on a network you have permission to snoop packets.

When I run this the output looked like:

crumb:~ root# tcpdump -n -l arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on en0, link-type EN10MB (Ethernet), capture size 96 bytes
19:10:48.212755 arp who-has 192.168.1.110 (85:70:48:a0:00:10) tell 192.168.1.10
19:10:48.743185 arp who-has 192.168.1.96 tell 192.168.1.92
19:10:48.743189 arp reply 192.168.1.2 is-at 00:0e:e7:7a:b2:24 19:10:48.743198 arp who-has 192.168.1.96 tell 192.168.1.111
^C

(To get it to stop, I pressed CTRL-C).

If you get a permission error, you may not be running the command as "root". tcpdump has to be run as root. You wouldn't want just anyone listening to your network, right?

After the header, we see these "arp who-has X tell Y" lines. Y is the host that asked the question. The question was, "Will the host at IP address X please respond so that I know your ethernet (MAC) address? The question is sent out as a broadcast, so we should see any ARP requests on our local LAN. However we won't see many of the answers because they are sent as unicast packets and we are on a switch. In this case we saw one reply because we're on the same hub as that machine (or maybe that is the machine running the command. I won't tell you which it is.) That's ok, because we only need to see one side of the question.

That's our data source. Now lets transform the data into something we can use.

First, let's isolate just the lines of output that we want. In our case, we want the "arp who-has" lines,.

tcpdump -l -n arp | egrep 'arp who-has'

We can run that and see that it is doing what we expect. The only problem now is that this command runs forever, waiting for us to stop it by pressing "CTRL-C". We want enough lines to do something useful, then we'll process it all. So lets take the first 100 lines of data:

tcpdump -l -n arp | egrep 'arp who-has' | head -100

Again, we run this and see that it comes out ok. Of course, I'm impatient and changed the "100" down to "10". However, that gave me the confidence that it works and I can use 100 in the final command. You'll notice there is a bunch of headers that are output too. Those go to stderr (directly to the screen), and aren't going into the "egrep" command.

Oh, you might be wondering why I use egrep instead of grep. It's a terrible habit I picked up many years ago. egrep supports regular expressions (wildcards) while grep does not. grep used to be considerably slower than egrep, which was pretty amazing considering how many more features egrep had. However that's no longer true and all grep's have about the same performance (in fact, GNU grep/egrep/fgrep are really all the same program, using a different name just sets various defaults). Anyway... as a result of this history my fingers always type "egrep" instead of "grep". For the purpose of this article, they have the same results. (and let me emphasize that the speed of grep has been fixed on all Unixes.)

So now we have 100 lines of the data we want. It's time to calculate the statistic we were looking for. That is, which IP address is generating the most ARP packets. Well, we're going to need to couch each host that generates an ARP and count them somehow. Let's start by extracting out the host IP address. They are always the 6th field of each line, so we could use:

awk '{ print $6 }'

to extract them out, but I'm lazy. You see, to determine that it was the 6th field took a bit of time. I'm too lazy to count, and that looked like it was "about the 5th word", so I first tried it with "$5". That didn't work. So I tried $6. Oh yeah, I need to remember that awk counts starting fields with 1, not 0 like sort.

Anyway, I'm lazy so instead I use:

awk '{ print $NF }'

The "$NF" means "the last field". Why isn't it "$LF"? That would be too easy. No, seriously, the NF means "number of fields". Thus, "$NF" means the field that is NF-fields in from the left. Whatever. Just remember that in awk you can type "$NF" when you want the last field on a line.

tcpdump -l -n arp | egrep 'arp who-has' | head -100 | awk '{ print $NF }'

So now we get output that is a series of IP addresses. Test it and see.

Now we want to count how many times each IP address appears in our list. There is an idiom that I use all the time just for this purpose:

sort | uniq -c

First we sort the data. Then we run "uniq", which usually eliminates duplicates from a sorted list (well, technically it removes any adjacent duplicate lines... sorting the list just assures us that the same ones are all adjacent). The -c flag counts how many repetitions were seen and prepends the number to each line. The output looks like this:

   ...
  11 192.168.1.56
   7 192.168.1.230
  30 192.168.1.111
   8 192.168.1.254
  21 192.168.1.91
   ...

We're almost there! Now we have a count of how many times each host sent an ARP. The last thing we need to do is sort that so that we know who the most talkative hosts were. To do that, we sort the list numerically by adding a "| sort -n" to the end:

tcpdump -l -n arp | egrep 'arp who-has' | head -100 | awk '{ print $NF }' |sort | uniq -c | sort -n

When we run that we will see the sorted list. It will take a while to run on a network that isn't very busy. On a LAN with 50 computers this took nearly an hour to run after hours. However, that was after the machine with the spyware was eliminated. Before then it only took a few minutes to collect 100 ARP packets.

On your home LAN with only 1-2 machines this command may take days to run. Hosts are required to cache the ARP info they gather so after a machine is running for a while it should be very rare that it outputs an ARP if the only machine it talks (on the local LAN) is your router.

However on a machine with 100 or so hosts, this will find suspect machines very quickly.

We now have a very simple tool we can use during an outage. This doesn't replace a mutli-thousand-dollar IDS system, or a good anti-virus/anti-spyware/anti-worm system, but it sure can help you pinpoint a problem when it is happening. Best of all, it's free and you learned something about shell programming.

__the end__

P.S. There are some areas of improvement that make good practice:

1. tcpdump outputs some informational messages to stderr. Is there a way to stop it from outputing those messages? If not, how could we get cleaner-looking output?
2. Turn this 1-line command into a shell script. Put this in your "bin" directory so you can use it in the future.
3. Can 'tcpdump' be programmed to only gather ARP "who-has" packets so we can eliminate the egrep? (check the man page)
4. Can 'tcpdump' replace the functionality of "head -100"? (check the man page) Is it the exact same thing?
5. On such a small amount of data, if grep was 10x slower than egrep would you even care?
6. awk is a complete programming language. Eliminate the "egrep" as well as the "head" using awk. Or discuss why we chose to do it in 3 processes instead of just letting awk do it.