Old 13th May 2014, 22:42   #1
milosz
Senior Member
 
Join Date: Apr 2006
Location: chicago
Posts: 123
Need help finding reason for rebuffering, silent gaps, etc

How to troubleshoot frequent buffering / disconnects?

I've been running SHOUTcast Server v2.2.1.109/posix(linux x86) on a virtual server for quite some time, has been pretty stable. Last 2 days I am getting a lot complaints from listeners of rebufferings, silent gaps and disconnects. I get them also, connecting via any of the broadband connections that I have available (Comcast, TMobile, AT&T) I looked at SC_SERV logs nothing much to see, just clients connecting / closing. Nothing is abending, server seems stable, bandwidth tests, packet loss and jitter tests to various endpoints from the server show zero packet loss and plenty of bandwidth up & down. Jitter is good to most endpoints, with a few showing high jitter.

Any suggestions? I don't think that DNAS is doing anything wrong, I suspect some kind of network issue - either with the IP stack on the machine or more general issues in the datacenter. (Note: I have rebooted several times and rebuilt the network config. No improvement)

What can I look at to pinpoint what I suspect is some kind of network issue?

Is there a possibility it could be a file system performance issue- SC_SERV's input buffer running dry as SC_TRANS tries to fill it but the file system can't move the data fast enough?

Note this is a mono stream at 48 kbps, and all files are likewise 48 kbps mono, rather small files. the SC_TRANS logs don't show anything out of the ordinary.
milosz is offline   Reply With Quote
Old 14th May 2014, 02:05   #2
Bryon Stout
Senior Member
 
Join Date: Feb 2011
Posts: 377
sounds like a bandwidth issue.

What is you uplink port size? and what is your monthly bandwidth limit?
Bryon Stout is offline   Reply With Quote
Old 14th May 2014, 06:53   #3
milosz
Senior Member
 
Join Date: Apr 2006
Location: chicago
Posts: 123
The problem has changed... not getting any rebuffering or dropping connections now, instead I am getting some short (under 1 sec) random instances of silence; the silence is inserted into the stream, there is no lost audio- here's an example:

SHOULD BE "...we will be working on this later tonight..."
What sometimes happens: "...we will be work[silence]ing on this later tonight..."

NOTES

Unmetered (unlimited) connection, 50 MB/s up & down; the datacenter has dual 1.2 Tb/s connection to Internet backbones (Cogent and Abovenet)

Uplink is 50 Mb/s.

iftop shows current bandwidth use is 18.1 Mb/s peak (including overhead)

I thought it might be related to disk performance, but iostat shows only 6% utilization.

I am baffled.
milosz is offline   Reply With Quote
Old 14th May 2014, 19:11   #4
milosz
Senior Member
 
Join Date: Apr 2006
Location: chicago
Posts: 123
Need help finding reason for rebuffering, silent gaps, etc

How to troubleshoot frequent buffering / disconnects/ short silent periods?

I've been running SHOUTcast Server v2.2.1.109/posix(linux x86) on a virtual server for a while, has been pretty stable. Last 2 days I am getting a lot complaints from listeners of rebufferings, silent gaps and disconnects. I get them also, connecting via any of the broadband connections that I have available (Comcast, TMobile, AT&T) I looked at SC_SERV logs nothing much to see, just clients connecting / closing.

It happens with the same frequency with 20 listeners or 220. Nothing is abending, server seems stable, bandwidth tests, packet loss and jitter tests to various endpoints from the server show zero packet loss and plenty of bandwidth up & down. Jitter is good to most endpoints, with a few showing high-ish jitter.

Any suggestions? I don't think that DNAS is doing anything wrong, I suspect some kind of network issue - either with the IP stack on the machine or more general issues in the datacenter. (Note: I have rebooted several times and rebuilt the network config. No improvement)

What can I look at to pinpoint what I suspect is some kind of network issue?

Is there a possibility it could be a file system performance issue- SC_SERV's input buffer running dry as SC_TRANS tries to fill it but the file system can't move the data fast enough?

Note this is a mono stream at 48 kbps, and all files are likewise 48 kbps mono, rather small files. the SC_TRANS logs don't show anything out of the ordinary.

NOTES

Unmetered (unlimited) connection, 50 MB/s up & down; the datacenter has dual 1.2 Tb/s connection to Internet backbones (Cogent and Abovenet)

Uplink is 50 Mb/s.

iftop shows current bandwidth use is 18.1 Mb/s peak (including overhead) - this is with about 375 listeners.

I thought it might be related to disk performance, but iostat shows only 6% utilization.

CPU utilization is about 15%
Memory utilization is about 85%

I am baffled.
milosz is offline   Reply With Quote
Old 15th May 2014, 19:35   #5
Bryon Stout
Senior Member
 
Join Date: Feb 2011
Posts: 377
What if any auto DJ program are you using?

What do you use to encode your mp3s with? I know that if they are not all encoded the same you will experience some weird issues.

At least back when I used WHMSonic.
Bryon Stout is offline   Reply With Quote
Old 15th May 2014, 21:14   #6
DrO
 
Join Date: Sep 2003
Posts: 27,873
try adjusting the network buffer options (either increasing the adaptive buffer from it's default to 5-10seconds) or change to a fixed buffer size and see if that helps. otherwise there's not much else i'm able to suggest.
DrO is offline   Reply With Quote
Old 15th May 2014, 22:30   #7
milosz
Senior Member
 
Join Date: Apr 2006
Location: chicago
Posts: 123
Found the cause of rebuffering!!!

Turns out it was a PERFORMANCE ISSUE with the overall Ubuntu server (for the most part.)

The server would bog down with I/O issues (mostly disk I/O) and just wasn't servicing the network as it should have been- even PING was often returning RTO etc.

The problem was this: This is a VIRTUAL Ubuntu instance, the hypervisor here is Zen. Hypervisors have their own routines for optimizing I/O, and so allowing Ubuntu to try and also optimize I/O (as it would do if the Ubuntu were running on metal) is counterproductive. The solution is to add the elevator flag to the grub.conf to change the IO scheduler from cfq to deadline. This makes Ubuntu treat I/O as near-realtime and makes disk and even network I/O much faster in this virtualized environment. Probably ALL virtual Ubuntu servers for streaming should be set up this way. One of the support guys at FDC suggested this and it worked like a charm. (I wish I would have known this.)

Also, the service provider (FDC) did something with the router which improved network latency and jitter somewhat, but mostly the improvement came from this "deadline" tweak.

Made a HUGE difference. Now everything runs as it should. Even the SSH shell is SO MUCH FASTER to use....

THANKS FOR THE TIP, THOUGH
milosz is offline   Reply With Quote
Old 15th May 2014, 22:34   #8
milosz
Senior Member
 
Join Date: Apr 2006
Location: chicago
Posts: 123
No DJ software

I'm just running the DNAS transcoder ( sc_trans) off a playlist. Not using Winamp or any kind of player or client at all. My stream is 48 kbps mono mp3 ; all the files played are also 48 kbps mono mp3, so the transcoder doesn't have to do any on-th-fly rate conversion etc
milosz is offline   Reply With Quote
Reply
Go Back   Winamp & Shoutcast Forums > Shoutcast > Shoutcast Discussions

Tags
buffering, connection problems, silent gaps

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump