...more recent posts
BitTorrent creator Bram Cohen today announced a deal with the MPAA under which DMCA takedown procedures for infringing content will be "expedited." But don't get too worked up when you read about this. At worst it is meaningless. There is just technically no way for them to stop it - this deal is sort of like securing airlines by prohibiting toe nail clippers. It might make someone somewhere feel a little safer, but it's not actually doing anything to address the underlying issue. I suspect this is another example of the industries (be it Recording or Motion Picture) failing to understand the technology. And perhaps Bram has even pulled off a small coup here, as trading nothing for a little legal cover sounds like a pretty good deal to make.
Tom helped me move the server - ash.datamantic.com - to the data center. I took the old server out, put in the new one, fired it up and everything seemed to be okay. I had put in the new network configuration back at headquarters. But then I tried to SSH into tulip (the current server located in a different data center,) and it wouldn't recognize my password. Weird. ifconfig seemed to indicate that ethernet was up and connected. And getting a response from tulip (even a rejection) further made me believe that things were working. But I couldn't account for the inability to log in.
But luckily (I am learning) I didn't waste tons of time trying to figure out what was wrong. I say luckily because, as I suspected, everything really was fine. I'm back home now I can ssh into ash with no problem. So everything seems to be working. I'll have to try to figure out the weirdness about not being able to get a secure shell going out though, but it's not a big deal at the moment.
So I'm a little behind schedule, but at least it has finally happened.
One of the main things I have my eye on is making it much easier for users to upload files to the server. Making a post or a comment is easy, but getting even a small binary file, like an image, is a few more steps, and getting a large one (say, over a meg) is ridiculous - you need to leave the browser entirely and go through a bunch of convoluted steps.
I am looking at the uploading issue from a lot of directions. I think that email is going to be an important upload channel, especially from mobile devices. And I'd like to see some integration with my desktop tools which for me would mean iTunes and iPhoto. I want to have a playlist in iTunes and an Album in iPhoto, and any song or image I put in them is automatically put onto the server. I actually have this working in iPhoto, except you have to click on a program in the dock when you want it to upload - but then it will put everything in that album onto the server and erase the images from the album.
But we really need to be able to upload larger files through the web upload interface. This works fine for things under a meg, but above that it craps out. This is because of the way Apache handles these file uploads. Lighttpd is another, newer, open source web browser that aims to fix this problem. And a few others that will be important to us as well. Seems like this is the web server to use (at least it will be when they get all the kinks worked out) if your site deals with a lot of large binary file uploads and downloads. Here's the Lighttpd homepage. And here's the explanation about large file uploads.
I'm going to still mainly run Apache, but I hope to have Lighttpd running as well for certain virtual domains that deal with its' sort of thing.
Okay. Now I've:
Installed: kernel-smp.x86_64 0:2.6.9-22.0.1.106.unsupported
Complete!
That should give me NTFS support although I have to admit to not liking that word 'unsupported' being anywhere near the word 'kernel'. Probably I'm just a too easily freaked out novice though. Here goes...
You learn something new everyday. It's not always what you want to learn though.
So I have the NTFS drive mounted on the Mac. And my idea was that if I can just get the Mac and the Linux box to connect directly to each other (over ethernet, not over 802.11b to the router) that everything would go much faster. After all their are gigabit ports on both ends.
I thought maybe I couldn't get it to work this way before because I need a crossover ethernet cable. I'm not sure that is true, since the Mac ports are supposed to be auto-sensing, but maybe that's only when connected to other Macs. Anyway I bought the crossover cable and got it to work.
Hallelujah I thought.
Except the copy still only went at 300 KB/sec. WTF? I guess the network connection was not the limiting factor. I wonder what is? I guess it must be the external drive that is the bottleneck. I'm surprised it can't do better than that though. This is a 2 drive RAID-0 connected over 400 Mb/sec Firewire to the Mac (and then over gigabit ethernet to the server.)
In any case, I just went back to doing it wirelessly because that way I can still be on the internet with the Mac while it is copying.
Not sure how this is going to work though since at this rate it will take a *long* time to make the copy. While I figure out what to do I'm just going to let it run (or crawl) in the background. Almost to 1 GB! Only 369 more to go. :-)
Well, HFS+ is supported in the CentOS 2.6 kernel, which was surprising to me, but NTFS (Windows partitions) are not! Crazy. What a pain in the butt. My system can see it, but it can't mount it unless I make unsupported changes to my kernel. Yuck. Maybe I'll just mount it on my Mac and copy it over, but that is going to be even slower. Damn.
Seems like Sun is really kicking butt lately. A year ago I wouldn't have believed it, but that just shows again how little I know. First they released a sub $1000 SunFire X2100 1U server that looks really sweet. And now plans have been revealed for a massive storage monster called Thumper:
The 4U high system will hold two dual-core Opterons and support up to 16GB of memory. A more unique part of the server will be Sun's use of 48 SATA drives.Holy cow. And the key to utilizing all that storage is a new filesystem, ZFS, that will be included in Solaris 10. ZFS sounds *really* amazing. The sort of thing that might make someone consider some really expensive Sun gear. Only now their gear isn't expensive any more. Lookin' good Sun.
Turns out you can just plug an HFS+ formatted drive into the server and it will recognize it. That's with CentOS 4.2 on the standard 2.6 kernel. I wouldn't have thought that would work. I should always remember to try the easy way first. You know, just in case.
Anyway, transferring 100 GBs over USB is not so much fun. Not sure if it's USB 1 or 2. Might have to sleep with the amazingly loud server fans on tonight which will be interesting to say the least.
But it does feel good to be loading it up at last.
Server just arrived. It is rather bad ass looking.
Man I have so many posts brewing but I'm just not sure when they will come out. I feel like everything has been on hold for so long, and now it is all starting to happen at once.
The new server is our way of playing a little part in this next phase of the internet. Disk space has plunged in cost, bandwidth - both fixed and especially mobile - have soared in speed, and new protocols - especially BitTorrent and RSS/ATOM - are allowing for new applications that will create a significantly richer media experience.
Here's one example. If you're a Mac user (Windows and Linux clients are on their way,) go download DTV. It's an RSS reader that pulls down feeds that contain video clips. You subscribe to different "channels" (which are like different websites,) and your DTV client polls them at a selective interval and downloads any new content. It's a program you just keep running in the background all the time, and then whenever you want you can open it up and there will be tons of new video clips that you watch right in the DTV program. And the downloads can themselves be torrents, in which case the DTV client is also a BitTorrent client - except the user doesn't see any of this, nor needs to know what any of these protocols are or how they work. Very very slick.
The DTV client looks and works much like iTunes. Click on 'Channel Guide' at the top of the left hand column (like clicking on your 'library' in iTunes) and you get a listing of different channels they think you might like to subscribe to. But the channels are just RSS feeds, so anyone can make a feed that can be shown (this is an open system - not a walled garden.) Just click 'Add Channel' at the bottom of the left hand column and you can enter any URL. Here are two to start with:
http://del.icio.us/popular/system:media:video
http://www.commonbits.org/rss/tag/daily-show
The first is a feed of popular video clips from the group site del.icio.us. The second are daily short clips from the Daily Show.
Now nothing here is completely new. We've been reading RSS feeds for years now. And downloading with BitTorrent as well. But it's never all been easy enough to reach critical mass. Now it is. And the way they have baked BitTorrent into this thing is just mind blowing. It solves the bandwidth bottle neck of being a content producer. And that means it levels the playing field significantly in terms of who can distribute the media.
What blogs have done for the written word, and what is generally called "news", which is much like what P2P networks did to the music industry, BitTorrent and RSS combos like DTV will now do to television and movies. It won't destroy them, but it will completely change everything, creating a lot of opportunities for new comers and a lot of disappointment for entrenched players who don't see it coming.
Server shipped. Should arrive on Wednesday.
Like most things I try to do this project is taking much longer, at every stage, than I ever anticipated. But today I finally ordered the server. Should ship early next week.
Dual Opteron server with 2 GB ECC RAM, and 12 400 GB SATA hard drives in a 3U Supermicro case.