SanFran MusicTech Summit

I'll be moderating a panel on new developments in streaming at the SanFran MusicTech Summit this Thursday, May 8th at the Hotel Kabuki. Our panel will start at 1:50pm in the Osaka Room (the downstairs room behind the Spring Room). With me will be:

John Richey - Wireless Music Delivery Expert, Apple
Greg Ogonowski - VP of New Product Development, Orban
Chris Grigg - Head of Standards, Beatnick
Tim Pozar - VP of Engineering, UnitedLayer

We're going to be talking about delivery methods. New codecs. Streaming to mobile devices. Internet radio hardware devices. How to determine if you really need a content delivery network. It should be real fun.

Here's a blurb about the summit:

The SanFran MusicTech Summit will bring together digital thought leaders from the San Francisco Bay Area, as well as from all around the country to the region which currently leads the way in innovating (both socially, and technologically) new ways of interacting with both music, and musicians. We will be working long term to help enable a sustainable, ongoing, Northern California based music and related technology market.

Register for the Summit here

EVDO, Wireless Performance, Radio Remote broadcasts and violating your terms of service

As I sit here in my Las Vegas Motel Room (the Best Western Mardi Gras, selected only on the basis of price and proximity to the Las Vegas Convention Center, where NAB is taking place) I'm thinking about how bad wireless performance is in general.

Right now, I'm typing over EVDO, because the hotel internet - powered by Lodgenet's StayOnline - is completely dysfunctional ("timeout connecting to network"). This is the same StayOnline that gave us so much trouble at the Marriott in Austin when we were trying to cover SXSW. I guess I should learn never to depend on the in-room wireless internet at most hotels/motels, because if the hotel is busy at all - the network will be unusable.

But, we have an EVDO card! We bring our own bandwidth with us rather than rely on the hotel internet, because that way we can always have internet access over "Sprint's EVDO Rev. A networks with data speeds up to 3.1 Mbps!" Only it doesn't work that way. In fact, these days, we're lucky if we get 500kb down. Here's what I get from the Speakeasy Speed Test:

Not too impressive.

You see the problem happens when there are too many EVDO users. And for Sprint (like Verizon), that means all the people that have their multimedia phones. And there are more and more of those out there all the time, fighting for the finite amount of bandwidth at each cell site.

We really saw this last weekend, when we webcast from Yuri's Night. At first, the webcast worked great. We had plenty of speed. But as all the geeks started arriving for the big party that evening, the network started getting slower and slower. By 8pm, that stream (from Stage 2) rebuffered so much it was pretty much unusable.

We were doing the main stage broadcasts from WayneCo's Bus which is equipped with a Motosat satellite internet uplink, which usually only gets about 256kb max for uplinks, unless you pay a hefty additional fee (which uses multiple transponders). So we didn't have enough bandwidth to stream both from the bus, and had to resort to EVDO for the other stage.

WiFi was also out of the question. With 5 SSIDs visible, the only reliable one was the backhaul network for the ticket booths, and not connected to the internet. The public internet was so overloaded that it often disappeared for minutes at a time. And only once were we able to maintain a connection, and that was before the event started. So we couldn't WiFi between our encoding gear at Stage 2 and the bus.

Everyone is always making promises of the happy wonderful infinite bandwidth wireless future. But it's still a way off. In a crowded situation, WiFi is about as useful as a CB radio, OK for really short distances, but for useful distances (200 feet or more) it falls apart. In this case, we could barely get 40 feet out of the WiFi base station in the bus to a remote machine. Sprint's EVDO works great sometime (4am in the morning in places where there aren't many users, for example) but lately in many different places we've used it, the service is over subscribed and slow, slow, slow.

Verizon's EVDO works just as badly as Sprint.

Wayne de Geere, who graciously provided his bus as our base of operations, has a Verizon EVDO, which worked about as poorly as the Sprint one. We chose the Sprint service because of Verizon's Terms of Service actually prohibit streaming audio and/or video and updating webcams and pretty much anything actually useful you'd do with their service. Sprint's restrictions are pretty much limited to things that violate the law.

Bottom line: we should have brought lots of wire. And run 600 ohm balanced audio from the stages back to where we were. Or installed wired ethernet connections to each stage. Or used something like a Marti SRPT 30 analog remote pickup unit, the technology that terrestrial radio broadcasters have been using for 30+ years.

Low tech, old fashioned, but tried and true.

Multicasting from the archives

Me, 5-Mar-99: ``Multicasting. It's close. And it is going to revolutionize internet radio.``

Man, I got that one wrong! Multicasting never caught on, not because the technology was bad but because there were never any business reasons for ISPs to enable multicast support in their backbone routers... there was no financial incentive for ISPs to support it; rather it was something that would ultimately cost them money to support it, and there was no way for them to effectively get paid for carrying the multicast traffic on their networks.

There was (and still is) no settlement model, hence no business incentive for multicasting. That's the main reason it never took off.

Remote Broadcasting Lessons Learned

I though it would be great for SomaFM to do a lot of live coverage from SXSW this year- after all, it's one of the biggest music and media festivals in the world. So I had some grand plans, many which have failed so far.

Plan number one:

Live webcams looking at the Six Street club area, as well as looking at the convention center and Brush Square Park. We have the cameras; we secured places to locate them in the Courtyard hotel next to the convention center. Alas, we didn't expect a total failure of the hotel's ethernet network system. And our backup plan didn't expect the hotel's wireless system to become overloaded and crash multiple times a day.

In fact it's a good thing that I have a Sprint EVDO card to get wide area wireless access to the internet, or I'd have no internet connectivity at all. But even Sprint's EVDO network is getting overloaded during peak hours (e.g. 10am-midnight local time). Today, it took about 30 minutes to upload 30 photos, when it should have taken 2-3 minutes. Plan number two: Live "Austin Audio" from above Sixth Street. The street sounds here in Austin have to be heard to be believed. And I though it would be really cool to do a live broadcast of the sound of Austin from the hotel a block off Sixth street. So I brought a couple of portable streaming encoders with us, and some stereo microphones to mount on the balcony outside the hotel room. Except that there were no rooms at the hotel with a balcony, and worse yet, no opening windows in the room! So scratch that plan.

Plan number three:

Quickly edit the podcasts, interviews, and band recordings each night on the laptop and upload before morning. The problem here is that we normally use ProTools for doing all our editing and audio production, but since ProTools won't work on OSX 10.5, I couldn't run it on my laptop. (SomaFM has one dedicated "audio production" machine that runs ProTools and also has the master music library on it; but this machine has to run OSX 10.4.x for ProTools.) I ended up installing a copy of Logic, but since I haven't used it much, there has been a learning curve that I hoped would have been faster. Also, some of the audio processing plugins we use with ProTools weren't licensed for use on a laptop.

Plan number four:

Broadcast the Bay Area Takeover party on Thursday live. OK, this one might still happen but given the way things are going, I'm not expecting it to work. We are still going to try. I used to scoff at the NPR guys when they'd send a crew of 5 people to SXSW to report on it. There are 3 people from SomaFM here, but two of those people (Merin and Elise) are also here on behalf of their day jobs and have to give 2/3rds of their time to that. What we needed was an audio production person to do the first passes at editing all the material we're recording. Maybe next time we'll bring an intern. :-)

In retrospect, here's what I think would have made things work a lot better:

Get a couple EVDO to WiFi+Ethernet routers, so we don't have to rely on internet from the hotels or venues at all. And bring plenty of ethernet cabling, as you may not be able to rely on the wireless networks.

Bring at least one extra laptop for processing pictures and audio.

Bring extra batteries for the digital recorders!

Arrive a couple days early to test everything... arriving 24 hours before the event starts won't give you enough time to get everything ready.

Have one person who's job it to just provide production support - and who doesn't go to the panels and conference itself - someone whose only job is to get the stuff posted and edited.

I wonder how this list will change in the next few days... we're basically winging it at this point.

Internet/Telecom in Iceland

I can't make a good judgement call about the net in Iceland, because I have a weak WiFi signal at the hotel, so I'm not sure how much of the slow net is due to the WiFi, the connection of the Hotel, or the connection of the continent.

I did notice that .is sites come up faster than US .com sites... and my photo upload speeds to Flickr seem to be in the 10-15k range a second at best.

However, the internet is everywhere here. Tons of cafés have free WiFi, and every café has people working on laptops in them.

Until I can test speeds at a few more places, I can't really attest to the speed of the net here.

The GSM network supports EDGE but because of the high data roaming charges, I haven't tested it. The cell coverage is pretty impressive, it works everywhere I've been since getting here.

San Francisco power outage, SomaFM outage

There was a power outage affecting downtown San Francisco today, which also caused an outage at SomaFM's primary datacenter, 365 Main. Note that we've been there for about 2 years now, and this is the first power outage that's affected us. They had another outage right before we moved in, due to a faulty fire alarm which cut power to most of the building.

Now, a "world class datacenter" is supposed to have all sorts of redundant systems in place. And they did. But a slightly unusual series of events proved that even with all that redundancy, things can go very wrong. Here's what really went down at 365main as far as I can tell:

365 Main, like most facilities built by Above.net back in the day, doesn't have a battery backup UPS. Instead, they have a "CPS", or continuious power system. What they are is very very large flywheels that sit between electric motors and generators. So the power from PG&E never directly touches 365main. PGE power drives the motors which turn the flywheels which then turn the generators (or alternators, I don't remember the exact details) which in turn power the facility. There are 10 of these on their roof (or as they call it, the mezzanine; it's basically a covered roof). These CPS units isolate the facility from power surges, brownouts and blackouts.

The flywheels (the CPS system) can run the generator at full load for up to 60 seconds according to the specs.

There are also 10 large diesel engines up on the roof as well, connected to these CPS units. If the power is out for more than 15 seconds (as I recall, I could be wrong on the exact time), the generators start up, and clutch in and drive the flywheels.

There is a large fuel storage tank in the basement, and the fuel is pumped up to the roof. There are smaller fuel tanks on the roof as well, with enough capacity to run all the generators until the fuel starts getting pumped up to the roof.

Here's what I suspect happened:

It was reported there were several brief outages in a row before the power went out for good, so I bet the CPS (flywheel) systems weren't fully back up to speed when the next sequential outage occurred. Since several of these grid power interruption happened in a row, and were shorter than the time required to trigger generator startup, the generators were not automatically started, BUT the CPS didn't have time to get back up to full capacity. By the 6th power glitch, there wasn't enough energy stored in the flywheels to keep the system going long enough for the diesel generators to start up and come to speed before switching over.

Why they just didn't manually switch on the generators at that point is beyond me. (I bet they will next time!)

So they had a brief power outage. By our logs, it looks like it was at the most 2 minutes, but probably closer to 20 seconds or so.

So it looks like the diesels did cut over, but not before the CPS was exhausted in some cases. The whole facility did not lose power I'm told, just most of it.

Here's the letter their noc sent to customers about this:

This afternoon a power outage in San Francisco affected the 365 Main St. data center. In the process of 6 cascading outages, one of the outages was not protected and reset systems in many of the colo facilities of that building.

This resulted in the following:

- Some of our routers were momentarily down, causing network issues. These were resolved within minutes. Network issues would have been noticed in our San Francisco, San Jose, and Oakland facilities.

- DNS servers lost power and did not properly come back up. This has been resolved after about an hour of downtime and may have caused issues for many GNi customers that would appear as network issues

- Blades in the BC environment were reset as a result of the power loss. While all boxes seem to be back up we are investigating issues as they come in

- One of our SAN systems may have been affected. This is being checked on right now

If you have been experiencing network or DNS issues, please test your connections again. Note that blades in the DVB environment were not affected.

We apologize for this inconvenience. Once the current issues at hand are resolved, we will be investigating why the redundancy in our colocation power did not work as it should have, and we will be producing a postmortem report.

Lots of companies were affected. There was a huge line to get into the data center. It was definitely the most people I've ever seen there!