...more recent posts
I remember the last time I made a large change to the system. About a year ago. The worst part was translating all the old posts into the new database. This time, thankfully, it's not seeming so bad. Maybe I didn't make as many changes.
Well, truthfully, I don't have the comments importing yet, so there may still be problems. But I just wrote something to grab all the posts on my page from the remote server ("oak") and put them into the new database here on my desk ("tulip".) I takes about a minute over this 56k dialup to suck it all down. Not bad.
I'm probably not as close to being done as it seems from the news that I'm already importing the old database. This is just for some testing so I can make sure I'm not missing any obvious problems just because I don't have many posts in the system.
The biggest finding so far, which is obvious but still worth noting, is that everything works really fast when it's all on the same machine. Wow. What a pleasure to not have to wait even one second for anything to happen. This makes the on line experience feel rather lacking in zip. I wonder how much of a difference a dedicated server will make?
Did some good work yesterday, and over the weekend, after last Friday's failing. Seems I need some down time after every few productive days. It feels like I'm not doing anything, but maybe that's not actually the case. It's interesting the way the mind works on problems.
It used to be that I slept very deeply straight through the night. Every time. Now that hardly ever happens. Lately my pattern has been (as best I can make out) to sleep for a few hours. Then I wake up after a first round of dreams. Lay in bed for a while unable to get back to sleep. After about 30 seconds my mind starts working on code again. I can't help it so I just let it go. Eventually I fall back asleep, and then after another dreaming episode I wake up again and start back in on the code. I think I had about four cycles last night. This is pushing back my wakeup time a bit in the morning. But on the other hand, I'm actually working during the night.
Yesterday evening I was sitting at the bar at aKa having a glass of wine, waiting for MB, and writing notes to myself about things still needing to be done in the new system. And then it hit me. Not only could I move comment page directory entries out of the main directory table, but I could just get rid of them all together. Without losing any functionality! It's easy to make huge breakthroughs when your original idea was so far off. Here are some numbers concerning this site which might illustrate why I'm so excited about this:
There are 4,379 entries in the directory table. These entries correspond to the 4,379 pages on this site.
But under my new system I'll only need 231 entries to categorize the same information (most of those 4,379 pages are threaded comment pages which, it turns out, don't really need to be in the directory.)
The subscription table is even worse. With one entry for every users subscription to every page we presently have 144,507 entries in the table (33 users X 4,379 pages.)
In the redesigned system this number would be 7,623. I will have to add one column to the table to make this happen, but still it's a major savings.
These are some startling results. We'll see if I can make it work. If so, then my fears about hitting the ceiling for how large this system could scale have been pushed back quite a bit. I'm going to try to get it minimally working today.
H. ordered one of the brand new (well, O.K., "speed bumped") dual 1Ghz G4 Powermacs today. 5 days ship time. I'm very curious how 10.1 feels on that compared to a 500mhz G3. Little faster I'd guess.
I'm pleased to report that I am just now concluding the work day having done precisely nothing. Starting out this morning I had hopes of perhaps accomplishing very little, but by late afternoon I began to sense the possibility for much much more. At 5:00 I had a brush with disaster - almost opening up the text editor to look at some code - but a quick trip to slashdot took care of that errant impulse. And from there it was smooth sailing. Zeros across the board. A perfect game. I think congratulations are in order.
Still in deep. Things are going well though (or at least they seem to be, given my lack of any formal testing ability.) Here's a few links I came across today while looking for other things:
Rebecca Blood (of Rebecca's Pocket) has a book coming out titled The Weblog Handbook.
Scoble reports on a brief demonstration of blogger pro ($5/month.)
Techno-weenie has posted an RFC on creating a "common XML-RPC API for weblogs." I'd support something like this but I wonder how useful it will be. The best reason to have so many different blogging (or knowledge management, or whatever) products is that they all work a little bit differently. I might try to support a common API, but not if I had to give up some of the stranger features I have built into my system. And if the structures of different systems are too different then the XML-RPC is going to be a mess. Or, in other words, the problem is here:
option (struct) -- extra weblog-specific options passed in the format parameter_name => parameter_value. Ideally, the options should be totally optional and the XML-RPC methods should function correctly without them.Ideally maybe, but at least in my case, the option struct would have to contain lots of crucial information (given the rest of the structure presented.)
Two pages of pictures of 76 Clinton St. progress.
Well, I'm sure you're dying for an update.
The new system (running locally on my production server) is almost usable. Note that this does not mean it is close to being done. At this point I can:
create pages
edit pages
view pages
post
add comments
edit posts and comments
None of these are tested very well. I'm still trying to figure a way to write a tool that will help me test (basically a tool that can simulate many different posts and edits from many different simulated users.) Right now in order to test I make a bunch of different accounts, sign in as different people, make posts, sign out, and then sign in as someone else and see if everything is OK. This takes a lot of time, and it is not very thorough. That's why everyone here has discovered so many bugs.
In addition to that testing tool (which I may not be able to write) I still have to enable the following:
admin
subscribe
upload
user
index
These should be easier than what I've done already. Post, edit, and the main script that draws the pages, as well as the one that draws comment pages, are where most of the changes are.
Then, when that is all done I have to write something to transfer the old database (I'm going to start with a different site than this one as the test, but it has the same database structure as this site so I should only have to write the translator once) into the new format. This is always tricky.
Could be done with just a few more full days work if I don't hit any snags.
Up early this morning. I made MB breakfast because I feel bad about how hard she is working on the new restaurant. I worked hard yesterday too, but it's different. We had a great dinner at Bond St. last night where I talked maybe too loudly about the dangers of "property rights fundamentalism" (as Lessig recently turned the phrase.) Actually I kept trying to drop the whole thing, but nonetheless conversation continued to circle back. All I was saying was that no sort of content protection will ever work as long as people have access to general purpose computers. But apparently that led me into some sort of dystopian fantasyland. Still, I'm not sure I'm so far off. There is a lot of content, and a lot of powerful people who own that content and want it to be secured. And if it's true (trust me, it's true) that there is no way to secure it while people have access to general purpose computers, then it stands to reason that those same powerful forces will try to go the legislative route and outlaw general purpose computers. My guess is they probably won't win, but I think it's pretty clear they are going to give it a shot. Notice that Valenti recently called people sharing (pirated) video content "terrorists" (last week in the NY Times, but I don't have a link.) That seems pretty calculated to me.
Maybe this stuff is already in the Patriot Act, I don't think anybody has even read that whole thing yet. We'll see.
Hopefully I can get another long day in today. I have to make some important structural decisions. This is where having been through the problem a few times before really helps. It sharpens your intuition. Or let's hope...
More mozilla strangeness: I can't set a cookie for mozilla 0.9.7 (OS X) from my local server (127.0.0.1)
IE 5 has no problem.
Getting some good work done today (finally!)
My friend J. convinced me that rewriting my code base wasn't necessarily such a bad idea. Especially if I've done it less than three times. Indeed, this will be the third time, so let's see if it's a charm like he suggests.
I'm not doing a complete from scratch rewrite. But I am going through every line of every script. Plus I have changed the database structure slightly to get rid of the scaling problem that threaded comments was going to cause.
I'm trying to add lots of documentation in-line this time. And of course clean up all the glaringly crappy code (at least to a regular crappy level.) But the real focus is on making the whole thing much easier to move from machine to machine. That means abstracting all the machine specific information out of the main code into configuration files. I'm also trying to abstract out as much HTML code as possible so that if I ever come to really care about standard compliance I can fix my good-enough HTML without going into every single script.
This new improved code base will be running on my home (production) server. When it stabilizes I will get the new colo server and move this new code to that machine. Then I'll move some other people to that machine and see what happens. Then if everything seems good I'll move digitalmediatree to the new machine and run things in parallel for awhile. Then finally I'll close this account and move digitalmediatree.com to the new server. Probably that is not too close to happening.
I haven't fully figured this out yet, but I will also try along the way to write something that will sync the contents of the production server database with the digitalmediatree database. I've never done anything like that before. I don't think it will be easy, but I think I can do it. If that works then I will provide a local copy of the system to anyone here who is running OS X. That might turn out to be pretty cool.
Christopher Locke is at it again. Burning down the world, that is. Inspirational.
Mozilla is getting aquafied (screenshot.)
AOL/TimeWarner is apparently in negotiations to buy Red Hat.
That would sure be interesting. But why would they buy a linux distro? A content company? Doesn't make any sense. They should buy a BSD distro (or just make their own) and then lock everything down. Of course I'd be happy if they went with Linux.
I'm discovering that my browser (Mozilla 0.97 on OS X 10.1.2) doesn't show up in referer logs. That is very strange.
Bruce Schneier on national ID cards:
I am not saying that national IDs are completely ineffective, or that they are useless. That's not the question. But given the effectiveness and the costs, are IDs worth it? Hell, no.
I've been busy. Had one interesting meeting I'll hopefully be saying something more about in the near future. The new restaurant opening keeps slipping slightly. Second week in February now. Oh well. Just lots of little problems; no show stoppers.
Not much time to write. Maybe you want to check out some new finds: quantumslip, and environy for some unusually high quality general purpose bloggage, and the all star team at satn for something a little more geeky.
Pretzel. Right.
I think maybe W. forgot his role, spoke up at some meeting, and Cheney had to give him a little smack down. "Remember George, you fell and hit your head while choking on a pretzel."
"Aw, jeez Dick, can't we make up something better than that?"
"You want some more frat boy?"
I saw The Fellowship of the Ring yesterday. I don't have too much to say about it. I thought it was a great if somewhat impossible film. Of course I wasn't going to really like it. Not like I really liked the books the first time through when I was young. But given what it was up against, the film did quite well. I just have no idea how it would go over if you don't know the books. Good cave troll anyway.
Gandalf was perfect. I thought Galadriel was too, although Alex thought she was stiff. I guess I can see what he means, but still...
Couldn't they have done it in 6 episodes but released 2 at a time in December for three years? The first could have been just up to Rivendell - with all the Tom Bombadil scenes added back in. The second would continue to the end of The Fellowship - but with a much more expanded opening council at Rivendel.
Probably The Return of the King could just be one movie.
I guess that's too risky (or expensive,) but there was a lot of material that had to be skipped. Still, given the actual time constraints, I generally agree with all the choices.
Sure makes me want to go to New Zealand.
Google is now indexing up to the minute news from over 70 sources. While traditional publishers, especially the New York Times, keep getting it wrong, Google leaps straight to the top of the on line news world. This is the front page of the news web. Thanks again Google. [via joho]
I've actually gotten this email more than once. It would be funny except the tone is so serious that I don't think it's a joke. I almost didn't post it because I don't want to make fun of him if he really believes this stuff, but he is sending these out to random people, so here goes:
From: webmaster@***.comWow. I'm thinking about sticking that last line onto the end of all my emails. Good luck Frank Young.
Date: Thu Jan 10, 2002 03:37:49 PM US/Eastern
To: *******
Subject: Time travelers PLEASE HELP!!!!!!
Anbei ist das Ergebnis des Antwortformulars. Es wurde versandt von
Frank Young (webmaster@***.com) am Donnerstag, 10. Januar 2002 um 21:37:49
---------------------------------------------------------------------------
message: If you are a time traveler or alien disguised as human and or have the technology to travel physically through time I need your help!
My life has been severely tampered with and cursed!!
I have suffered tremendously and am now dying!
I need to be able to:
Travel back in time.
Rewind my life including my age back to 4.
Be able to remember what I know now so that I can prevent my life from being tampered with again after I go back.
I am in very great danger and need this immediately!
I am aware that there are many types of time travel, and that humans do not do well through certain types.
I need as close to temporal reversion as possible, as safely as possible. To be able to rewind the hands of time in such a way that the universe of now will cease to exist.
I know that there are some very powerful people out there with alien or government equipment capable of doing just that.
If you can help me I will pay for your teleport or trip down here, Along with hotel stay, food and all expenses. I will pay top dollar for the equipment. Proof must be provided.
Also if you are one of the very few beings with the ability to edit the universe PLEASE REPLY!!!
Only if you have this technology and can help me please send me a (SEPARATE) email to:
Robby0809@***.com
Please do not reply if your an evil alien!
Thanks
I pointed to something similar a long time ago, but this one is even better. Night time picture of the earth from the space station. [via FMH]
Did I mention this before? It's been the number one most helpful OS X tip for me. Launch BBEdit from the terminal with:
sudo open -a "BBEdit Lite 6.1 for OS X"
(Yeah I'm cheap and haven't bought the full version even with the big sale.)
This launches BBEdit with root permissions allowing me to open and save files into directories I otherwise wouldn't have permission for (like the main webserving directory.)
Looks like Matthew Rossi worked through that depression, or writers block, or whatever. The ideas are certainly flowing now. Don't click unless you've got some time. And the correct neurological processes enabled.
How many times has man risen out of barbarism and tried to forge a civilization? The gap between the rise of the modern human and the rise of the first known culture, that of ancient Sumer, is tens of thousands of years at the most conservative guess. Was it merely a case of inertia to overcome before the slow march forward could become the torrential changes we have grown accustomed to, or is there truth to the various stories that say before the nations familiar to us, there were others known not today?
Doubleclick gets out of the targeted advertising business. You know, I really feel bad for these guys.
Oh, no, wait...
Cam has an interesting thought on the possibilities for anti-spam services. But what about a service that gives you an email address - very cheap but not free. Then anybody with an email account on that system also gets a central email address where they can redirect any spam they do get. These spams are then used to create a master profile. Every incoming email for every different account passes through the filter created by aggregating all the spams received by everyone on the system. Any incoming mail that matches anything in the central deposit is thrown out. You'd still get spam, but if the system grew large I would think it would be very infrequent. With enough people it would almost always be the case that someone else would have gotten that spam first. My guess is this might be very good to perfect at not producing false positives, while still being pretty good at stopping spam. And the real problem with spam filters is that you don't want false positives (you don't want even one in a hundred real messages deleted before you see it.)
Whoa.
TOKYO (Reuters) - Japan's Minolta Co Ltd said on Wednesday it had halted new product development for APS (advanced photo system) cameras and would focus its resources on the fast-growing market for digital cameras.I guess that whole worse than 35mm film quality thing wasn't such a good idea after all. I assmue they're not halting production on those more traditional film cameras as well.
Got my new memory. I'll be curious to see what kind of performance difference there is in OSX with 512megs instead of 128megs. I'll report back. (And I'll repeat my plea: memory is cheap, cheap, cheap. Buy more. If you don't have at least 256megs of RAM you should definitely buy more. 256megs for an iMac costs around $60 and will make your computer much faster.)
No new PowerMac towers? They've got to come soon because that top of the line iMac makes the present towers look a little underpowered and expensive.
But I think the iMac is a homerun. $1299? Sure that's expensive for the entry level model - but that thing is powerful. Really powerful. Hopefully I can convince my friends Virginia and Steven to buy one.
The biggest surprise for me was the new top of the line 14 inch iBook. I definitely didn't see that one coming. $1799. Makes it kind of hard to justify the TiBook. This is just like the iMac/PowerMac problem. The consumer machines are so rocking that they will take a bite out of the professional lines. So I think it won't be very long (way before MWNY in July) that those machines get bumped up too.
I was really expecting a "one more thing..." release for the towers. I thought they'd be 1.2 Ghz, 1.4Ghz, dual 1.4Ghz G4s. The fact that this didn't happen makes me think that the G5 is coming, but wasn't quite ready. Look for a special event by the end of the month, or maybe a MacWorld Tokyo release. I had made H. wait to buy her new tower, and she's a little disappointed, but what could I do? It might have happened today. Soon hopefully.
So while it wasn't "way beyond" the rumor sites - frankly it wasn't anywhere near what some people were expecting - I think the iMac is right on. Priced well. Beautiful. And sure to be a big seller.
Lots of dreams last night. At one point I was hanging by both hands from a cable stretched between two high points over a river far below. I was way out in the middle and very unsure I could make it to either side. I kept thinking 'this can't be right... how do I get out of this?' And then I went through the whole 'maybe I'm dreaming?' thought process, and determined that, indeed, I was dreaming. But I don't think I woke up. The dream just switched to something else. So, again, I'm close to what I understand to be lucid dreaming, but it doesn't work out quite like I expect. I only gain a little bit of control, like I can nudge events one way or the other. But I can't just decide to fly around, or talk to old friends, or... um... anything else.
Later - and yes, I know it's ridiculous to dream about such things, but it has been on my mind - I was dreaming about Steve Jobs' keynote speech at MacWorld. This actually happens in a little over an hour, but I saw it last night. There was a big image of a chip and I kept looking at it and at first it said G5 and then I looked again it said G4, and then back to G5. That's the sort of clue - text that keeps changing when you look away and then back again - that is supposed to remind me that I'm dreaming, but I missed it. Seems like I can realize I'm dreaming much easier if I'm in a tight spot and the realization will serve to get me out of it. Anyway, then it changed and I saw a very tiny Mac that was somehow hooked up to a Nintendo GameCube. Or it was a Nintendo Gamecube. Yes, I know, my life is fascinating.
And then later I was hanging out with my friend Sarah who is still away out West. I miss her. When are you coming home Sarah?
Bare Bones Software has a special deal on BBEdit 6.5 from now until Jan. 11. $85, or $44 if you registered any past copy (even the old bundled ones that came with Dreamweaver.) Nice.
Alex took a beautiful picture of the midtown skyline from Central Park to go with his epiphany post.
This message board has answers to a lot of my OSX questions.
Well it's frustrating, but that makes any small amount of progress fun. I've spent the entire morning trying to get mod_rewrite to work correctly. That's a module for the apache web server that allows (the way I am using it) for me to send every request - regardless of path - to one central php script for execution. All my sites are built around this ploy so that I can have regular looking URLs that all call the same script which assembles the apparently static pages on the fly from the mysql database. Without mod_rewrite I'd have to have URLs like:
www.digitalmediatree.com/page.php?path=/jim/weblog/
which would call the php program page.php and pass it a variable (here $path) with the page path I want. Besides being ugly, some search engines don't like the '?' in the path and won't follow such links. With mod_rewrite I can accept URLs like:
www.digitalmediatree.com/jim/weblog or
www.digitalmediatree.com/treehouse/comment/1143 or
www.digitalmediatree.com/arboretum/archive/2001/12
and have each request actually invoke the same php script (like page.php) which can then parse the request path (/jim/weblog or /treehouse/comment/1143) and figure out what page to call from the database. These two examples produce the same result - either calling the central script in the URL like the first one and passing the page variable to it on the end of the URL, or using mod_rewirte to invisibly force the calling of the central script and having the script extract the variable from the request_uri environment - but the later way yields better looking URLs, and that turns out to be pretty important on the web.
Anyway, like most problems this one turned out to be fairly simple. But that doesn't mean the time to find a solution is short. Apple ships Apache with the 'AllowOverride' directive set to 'none'. This foils the use of .htaccess files for mod_rewrite rules. Changing the httpd.conf line to 'AllowOverride all' solves the problem.
The rest should be fairly easy but boring work of changing some file system specific paths from what they are on the linux server to what they need to be on my iMac. If I was a better programmer I would have had all of that stuff in some sort of configuration file so I didn't have to go back through all the individual php files. I'll try to implement that more tidy centralized system this afternoon.
Here's a couple of very off color balanced photos of 76 Clinton (Alias Restaurant.) These were taken on 12/27/2001, so work has progressed slightly since then. I believe 2/1/2002 is now the tentative opening day.
My local email client was not checking my digitalmediatree.com email account for the last few days (operator malfunction.) I think I'm caught up now. Sorry if you were temporarily ignored.
It's weird how Apple has root disabled by default. That threw me a little. In any case, I finally have apache, php, and mysql all running locally on my little iMac. 512 megs of additional RAM are on the way. I can't tell you how great this is. Being able to develop on my local machine - without FTPing every change to my server over a 56K dialup - is about the best thing I can think of.
Hopefully by the end of today I'll have this site replicated at home. And of course the ease of setting this up under OSX is making me think of strange distributed scenarios where every user has the full system replicated on his/her home computer, and everything is synched through a central server. That way traffic spikes (ha!) could be handled by a round robin configuration of all the users. I just wish that internet access was developing in such a way that casual serving was possible. Unfortunatley it seems like the big providers want you to consume but not serve. Still, there are always ways around.
One interesting thing I noticed is that even on my dial up (upstairs, not down in the office) my web server is accessible from the outside. Granted the IP address would change every time I got disconnected, so it's not very practicle, but I wouldn't have expected it to work at all. When I get something up here I'll post the IP address and you can try to hit my iMac from where you are. Because, you know, edge of the seat excitement is what we're all about.
I miss lemonyellow.com and sevencrabrangoon.com.
Of course what I really need is a $500 rechargable fleece jacket that heats itself.
The front page of the Apple website is counting down the days until new products are announced at MacWorld San Francisco next Monday. They started two days ago with a big headline reading: "7 days until Macworld 2002 - This One is Big. Even By Our Standards." Then the next day they changed it to "Count the days. Count the hours. Count on being blown away." And then today, with five days left, Apple really stirred things up with the headline "Beyond the rumor sites. Way beyond." The message boards are reaching critical mass as the Mac fanatics outdo each other speculating about what could be so big. I have to think the people looking for dodecahedron shaped G5 towers starting at 2 Ghz are going to be a little disappointed.
Strange. I hadn't been able to access my canon G1 digital camera since I upgraded to 10.1.2, but now I've discovered that it will recognize the camera if the AC power to the camera is not used. This doesn't make sense to me, but I'm happy to have access to my pictures again.
No, I'm not a very good dancer. That didn't stop me though. After a great night at AKA we staggered down Orchard St. to someone's loft where Nicholas - I mean, Big Jimmy Fingers - was spinning from 2:00 until ?. The whole place was covered in CDs. I mean the walls, the ceiling, the whole place. Covered in CDs. Nice.
And like I said, I'm not a good dancer. It takes a lot to get me out there. But it's fun when it happens. Full on ecstatic arm waving dancing. It's theraputic. Somehow we ended up with Bill's bottle of Hungarian Tokaji, which probably isn't the best thing to be swigging to the super heated disco beat, but that's what happened. Eventually (where eventually equals this morning) you pay for such transgressions. I was having such a good time I can honestly say I was nonplussed by the small asian girl in the front who would not stay inside her dress. Yeah, yeah. That's just what happens when it gets late. I completely forgot myself, which I guess comes to about the same thing as finding yourself. Your real self. Or one of them at least.
After that it gets a little fuzzy. Of course we wound up at Barramundi. That's where MB and I met, and in the end, that's familly. I think we were doing tequila shots with Erica, but I'll have to check with Tony to be sure. Thinking of a master plan, this ain't nothing but sweat inside my hand. Some day I'll write more about that place. It's special.
This morning seems like the best day ever. Crystal clear skys. Warm. I walked up to Russ & Daughters to buy bagels and loxs with my jacket open just basking in the bright winter sun. I've got a good feeling about this 2002 deal.
From metafilter comes my (probably) last post of the year.
Imagine that the history of the universe is compressed into one year—with the big bang occurring in the first seconds of New Year’s Day, and all our known history occurring in the final seconds before midnight on December 31. Using this scale of time, each month would equal a little over a billion years. Here’s a closer look at when important events would occur when we imagine the universe in one year.The rest of the months are condensed on the linked page, but here's the expanded December calendar for our thought experiment year:
I love that the final 10 second New Year's countdown brings us from the building of the pyramids all the way to the present. At this pace we should have the galaxy populated by the end of New Year's Day. Let's get to work.
Apparently I'm the only person in the world who likes this holiday. Fine. So be it. I know you'll be celebrating anyway, so aren't your protestations a bit hollow? "I just hate staying up late into the night with friends drinking champagne." Huh?
My friends this is it. This is the great major holiday. Unencumbered by religious significance or family duty. The one where alcohol, rightly, takes precedent over the large dead animal featured on the lesser holiday tables, over trinkets of commerce exchanged to and fro. For truely, who needs such foodstuff? Who needs material gifts? I tell you, good friends and fine bubbles are all that are required for life. For the good life. Drink deep. Kiss long. Float up into the heavens. This is what ever happens, yet our earthly bodies do not always get to partake. Don't scorn your chance. That decision might speak of regret.
But what do I know? As Frank says, I'm just a high class wino, and of course the first part is debatable. Or just wrong. Still, we'll be drinking the good stuff. The new restaurant isn't quite ready yet, so we'll be at AKA from 8:30 on. Stop by if you're on the gLower East Side. Everybody is welcome. Leave the scowl at home.