Troubleshooting slow Internet and unusual upload bandwidth usage

A friend of mine who owns a small accounting company of 4 employees, called me asking for help because their network was incredibly slow. They were having trouble opening Internet pages, sending emails (especially with attachments), and their phone system (VOIP) was getting very choppy and distorted.

They've had the same cable Internet service for 8 years, and on it, they run a small business SIP system for their phones (supported and managed by a 3rd party).

Their network topology is simple:

Cable modem connects to a Mikrotik router, in which the following connects (no switch): SIP system, File Server, 4 Desktop PCs, a Printer/Copier.

The first thing I told my friend, was to contact his ISP to check the cable modem's signal quality and to check his bandwidth usage. I explained to him that based on the price he paid, and comparing with the latest business Internet plans offered by his ISP, he was probably using a 12Mbps down/1Mbps up connection, which is far from enough to cover for the typical usage of 4 users and a VOIP telephony system. As a matter of fact, his ISP confirmed my suspicion (he was using a 12/1), and strongly suggested he upgrades his plan to at least a 30Mbps down/30Mbps up connection, which he did.

However, more disturbing was the fact his ISP also confirmed his upload bandwidth was completely saturated, and had been so for the last 4 days, 24/7, and totalled an amount of 50Gb of data transfer in those 4 days, compared to the usual 4Gb he does monthly. They typically only send occasional PDF attachments by email and browse government websites, which is hardly enough to generate that much upload traffic in that small amount of time.

Based on that, I suspected that someone or something was using his network and Internet bandwidth to upload mass amounts of data to the outside, and urged him to check all of his PCs and servers to see if he could find where it came from.

He had his IT tech come and check all of his PCs, updated his antivirus definitions, updated Windows, scanned for ad/spyware and several other things but didn't find the source of the bandwidth hog.

He then called in his VOIP provider to check his router and VOIP system. The 3rd party sold him a Mikrotik router which it manages from the outside and that router acts as the only line of defence for his network (as typically does an Internet-facing router). The tech did several spot checks, and ended up flashing and updating the firmware of the router (which was a bit outdated), and then secured a few things that were enabled, such as reverse DNS.

About at the same time he rebooted the router to apply the firmware, the ISP confirmed that the upload traffic had stopped and the VOIP tech was happy to say that his changes fixed the issue and were "obviously" the most plausible cause. His rough explanation being that there were a ton of intrusion attempts and they were overwhelming the router, and that the upload traffic (which totalled 50Gbs in 4 days if you remember...) were caused by all the replies sent back by the router.

A few things didn't make sense to me then and still don't, and I'd like your opinions on the matter to confirm or deny my doubts:

1. I thought that in order for TCP/IP traffic to be considered upload traffic (by the ISP), the source of the request had to come from the inside, and had to go to the outside (Internet). Even if the router replied to an inbound request, the reply being outbound would still be part of the same inbound request, and so would could as download traffic for the ISP. At least that's what I thought I understood. So knowing that, it would not be logical to say that allowing reverse DNS queries could contribute to upload traffic, right (the 50Gb/4 days) ?

2. Whether or not point 1 is what I thought or not, how in the heck could it total 50Gbs of upload data ? I mean, what kind of intrusion attempt (even one launched using a daemon or a script) could generate that much data in 4 days ? It may slow down the router by overwhelming its processor and memory, but how could it cause so much data to be transferred ?

3. My initial assumption was that this kind of upload traffic was the kind I would be expecting to see from someone who is seeding large torrents 24/7 (knowing they only had a 1Mbps upload limit, which makes for roughly 125Kb/s) or was transferring huge files continuously. I can't explain that kind of usage by anything else than a large or continuous file transfer.

What do you think cause have caused that much upload traffic in that much time, and what do you think of the explanation of the VOIP tech ?

 

Before I get into single aspects: It's no surprise to me that 1Mbit upstream in a 4-man company is maxxed out most of the time (during work hours)

1) Even if the routers logic counts IP reply packets as outbound traffic.
Technically its outbound traffic which uses the upstream bandwidth regardless what upload statistics of a router say (which indeed mostly dont include ICMP and basic TCP overhead traffic)
On the other hand saturating a whole upstream channel of 1 Mbit with echo replies for example would imply that you would need for example TWO ping commands of

Code:

ping -l 65500 -t google.com

running in separate processes

or if google cuts off the size of big ICMP packets then six of

Code:

ping -l 20000 -t google.com

That would be enough to use up 125kbyte/s upload bandwidth.


2) If you calculate the duration for 50GB over a 1Mbit line you will get:

Code:

[I]Results:
Speed: 125 KB/s
Size: 50 GB
128 Hours, 09 Minutes and 33 Seconds[/I]

That means its totally possible if your upstream is maxxed out over 4-5 days including night time.

3) Torrent seeding as you said comes to mind as well.

4) for VOIP traffic a rule of thumb is 100kbit up+down per concurrent call

My advice would be to buy a cheap managable switch set up a truncate port
and check with wireshark what traffic is going through the upstream at NIGHT times. That way you can identify a persistent connection which may occupy the upstream.
The hard-nosed method would be to disconnect single network devices over individual nights to check which device is causing a certain continous upstream. The next day the ISP should be able to provide if upstream did go down or not. This is a rather impracticable solution though.
So best would be to get an independent guy who the boss of this company trusts, maybe someone of his friends or family, who knows how to use wireshark in a switched environment. (I dont know if the integrated switch of the router can configure trunc ports).