mike at perusion.com
Sat Apr 4 16:53:57 UTC 2009
Quoting Jon Jensen (jon at endpoint.com):
> On Sat, 4 Apr 2009, Mike Heins wrote:
> > The back-end is the part that IC needs to support. The library itself
> > could be easily changeable on the consumer side. We don't much care
> > about that part except, as you noted, for the demo. And for one other
> > thing -- form widgets. Those are used in the admin extensively, and we
> > need to select something they can support.
> > For instance, I don't know what people are using as servlets for the
> > data. If you are using regular Interchange, that is heavyweight and not
> > scalable. If you are using anonymous database access via a connector,
> > that is (probably) not secure. If you are using anything else, it would
> > be nice to know what that is so we can learn from it.
> Most I know of are using straight Interchange in PreFork mode.
> Backcountry.com uses a custom MVC subsystem but it's still within the
> Interchange daemon in PreFork mode. The main optimization they use it to
> prebuild static files (JSON or HTML) which are fetched directly, with no
> Interchange involved. That allows them to use a CDN (Akamai or Limelight)
> to reverse proxy the content, so very few of the requests even hit their
> servers. Their http://www.steepandcheap.com/ and related "One Deal At a
> Time" sites use these techniques extensively.
> If a CDN is out of the budget range, a poor man's in-house CDN can be
> built with Varnish or nginx, with good results.
That's my point. There is a significant territory between the large massively-
accessed site and the site with so little traffic it doesn't matter
what you do. Interchange has only a few sites with backcountry.com
traffic and we are never going to generalize solutions for them.
But there are many sites with between 50,000 and 500,000 visitors per
month. During peak access times it does matter what they are doing, as
much for perceived performance as system load.
> > My idea is to standardize on DBI sessions, and provide a read-only mini-
> > session interface that doesn't require evalling the potentially-huge
> > blob. It would provide a row-based fetch for the session that would have
> > the minimal info
> > catalog,username,admin,superuser,groups,login_table,time plus anything
> > else you wanted to add (lname,fname,etc.). This would be specified and
> > automatically handled by SessionDB.pm at session store time. (Read would
> > be a separate function.) So you would set up access via regular
> > heavyweight IC accesses, and be able to check access permissions via the
> > read-only interface.
> > As part of enabling the Ajax back-end, you would specify tables that
> > could be queried and fetched, and you could allow access to certain
> > tables based on username (i.e. transactions=username,orderline=username,
> > userdb=username).
> I guess I would suggest that you try some benchmarks before you go too far
> down that path. For all the work, I don't think that will provide that
> much of a performance boost. Interchange itself is the slowest thing in my
> experience; sessions certainly slow things down too, but whether you fetch
> the whole session or only a part, you're still hitting the database and I
> don't think you'll achieve much more than a modest speedup if any.
That's the whole point. The servlet isn't Interchange-based, but uses
the data written by Interchange on session initiation. I am not talking
the admin here, that is another story and doesn't really need to be optimized.
> Our experience at Backcountry.com with DBI sessions has been that they
> quickly become a nasty single point of failure and a load bottleneck when
> traffic picks up. MySQL with MyISAM is unusable with any significant
> traffic due to its insane table-level locks taken during any write. MySQL
> with InnoDB has serious bloat problems and no online vacuum. Postgres has
> less serious but still real bloat problems, though with 8.3 and HOT, the
> bloat is much more manageable.
Then what you do is provide a separate DBI write for the minimal parameters,
with the blob still stored in a file.
> But in any case, Backcountry years ago passed the point of being able to
> survive with a single session database, so we developed a new
> multiple-session server sharding-type setup. If anyone else would be
> interested, I can provide a patch. It's very specific to their setup and I
> wouldn't recommend it unless you are planning to throw 3+ dedicated
> servers just to Postgres sessions, which is why we haven't released it.
> The most efficient way to scalability is to remove entire classes of
> resource access: database, session, or app server access altogether.
Of course. But a lot of that is unnecessary or impractical for much of
Interchange's user base. Face it, large sites don't often use
Interchange, and they are usually going to be doing custom solutions.
The one-server or several-server system is the vast majority of IC sites.
Those can benefit from some level of efficiency that allows them to
Ajax-enable their sites.
> > The admin page universe is different from the user side, so it should
> > not interfere.
> Except that many people have customized their admin to use Ajax, the place
> it first became acceptable because the supported browser universe was
> smaller for admins than for the general public.
We have done that too. But nothing we would do would invalidate that,
would it? The vast majority of places I see it happening are either 1)
in the table-editor widgets or 2) in pages like order.html or item.html
which are the ones that people override and customize.
> > It is kind of sad that no one has contributed back. That is a recurring
> > theme in the site design area, sigh. I only use our own little
> > proprietary things, with Prototype.js. But with a new project that
> > requires some more, I am looking to try and do that contributing back. I
> > suppose I could just be like everyone else and hack it in without
> > considering how to make it more general.
> Everything I've been involved with has been extremely specific to the
> client or their architecture, and we've seen no way to abstract it out.
> I've never had a chance to do Ajax work on a brand new site. If you're
> doing that, that's a great opportunity.
> Hmm. This is all interesting, but again I wonder if we should be
> discussing this on interchange-users. So few people get to participate
> here and it doesn't show up in search engines.
Copied to that as well.
Perusion -- Expert Interchange Consulting http://www.perusion.com/
phone +1.765.328.4479 <mike at perusion.com>
An amateur practices until he gets it right. A pro
practices until he can't get it wrong. -- unknown
More information about the interchange-users