Interchange HOWTOs: A Collection of Recipes

This documentation is free; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

It is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

Abstract


1. Interchange
1.1. Core
Controlling access to pages
Implementing banner ad display and rotation
1.2. Databases
Counting rows or number of items in a table
Finding inconsistencies in Interchange table definition files
Modifying text source files and applying changes
1.3. Demo catalogs
1.3.1. "Tutorial"
Providing "quantity" field on the basket page
1.3.2. "Standard"
Overriding standard Admin UI pages with per-catalog custom versions
1.4. E-mail
Checking for syntactical validity of e-mail addresses
Optimizing e-mail delivery with a custom Sendmail routine
1.5. Forms and form submissions
Defining mv_metadata width and height for HTML textarea
Testing for errors in form submissions
1.6. Ordering
Forcibly adjusting quantities of ordered items
1.7. Searches
Avoiding search strings in HREF= specifications
Creating custom search routines for use in mv_column_op
Supporting AND, OR, and other advanced search specifications
1.8. Server Process
Adjusting timezone
Dumping complete configuration ($Vend::Cfg)
Testing configuration, starting, reconfiguring, stopping
1.9. Sessions
Expiring
Reading and writing user sessions
Storing sessions in MySQL
1.10. Tax / VAT
Implementing basic, country- and category-based tax/VAT scheme
Implementing user-based tax/VAT scheme
1.11. User database
Searching through UserDB
2. HTML
Validating (X)HTML markup
3. Web servers
3.1. Apache
Logging to per-vhost logfiles
Using rewrite rules to avoid /cgi-bin/ in URLs
4. Databases
4.1. PostgreSQL
4.2. MySQL
Fixing constantly-failing SQL statements
5. Unix
Monitoring log files
6. Perl
Making Perl hash modifiable without overriding original values

1. Interchange

1.1. Core

Controlling access to pages

There's a built-in Interchange way to control access to pages served from the PageDir directories.

If the directory containing the requested page contains file named .access — and that file's size is zero bytes — then access can be "gated" in one of the following ways:

  • If the file .access_gate is also present, it will be read and scanned to perform page-based access decision. It will also override any other method.

  • If the variable MV_USERDB_REMOTE_USER is set to true value, any user logged in via UserDB will receive access to all pages in the directory.

  • If the variable MV_USERDB_ACL_TABLE is set to a valid database identifier, the UserDB module can control access with simple ACL logic. If MV_USERDB_REMOTE_USER is set, this capability is not available.

.access_gate file format is as follows:

PAGE_NAME: CONDITION

PAGE_NAME is the name of page to be controlled, and the page suffix (such as .html) is optional.

Page name of "*" can be used to set a default permission, but globbing is not possible ("ind*" will not match "index"). The default permission has lower precedence than any rule that matches the page name explicitly.

The CONDITION is a simple true (or Yes) or false (or No) value which may come, and in fact often comes, from interpolated ITL code. The implicit default permission is No, so leaving the default entry unspecified results in all non-matching users being denied access.

Here's a very simple example:

pubview: Yes
*: No

Here's an example that includes ITL interpolation and is verbose enough not to require additional explanation.

foo.html: [if session username eq 'flycat']
          Yes
          [/if]

bar:      [if session username eq 'flycat'][or scratch allow_bar]
          Yes
          [/if]

baz:      yes

*:        [data session logged_in]

Implementing banner ad display and rotation

Banner display in Interchange is implemented using the banner tag. There are two terms which need to be understood first:

  • Banner display - term referring to general banner display. Actual code behavior might be affected by the category or weighted attributes passed to the tag.

  • Banner rotation - term referring to selecting exactly one banner, if there are multiple banners defined in the banner database field. Those multiple banners are displayed in sequential order, with a separate counter kept for each client.

Additionally, we need to clarify the difference between categorized and rotated display.

Categorized display (mostly used in combination with weighting) makes all banners matching a specific category to be displayed the appropriate amount of time (adding weight gives a banner more visibility).

Rotated display makes multiple banners from the same banner database field to be selected in sequential order, for each client separately.

Both categorized/weighted and rotating techniques can be used together.

For the banner tag to work, you will need a banner table. Here's an example:

code	category	weight	rotate	banner
t1	tech	1	0	10% tech banner
t2	tech	2	0	20% tech banner
t3	tech	7	0	70% tech banner
tech	tech	0	0	tech banner, categorized, non-rotated
tech	tech	0	1	Tech rotated 1{or}Tech rotated 2
a1	art	1	0	10% art banner
a2	art	2	0	20% art banner
a3	art	7	0	70% art banner
default		0	1	Default 1{or}Default 2{or}Default 3

You might notice that the fields names and values in the tables above are not properly aligned. This is an unfortunate nature of tab delimited files.

To minimize the chance of confusion, you can download properly composed banner.txt.

Field descriptions follow:

  • code - key for an item (of course, the key has to be unique).

    In a default non-weighted display, the category attribute is expected to reference this unique code.

    In a weighted display, the value of this field is not that important because the choice is made by selecting database rows regardless of their code.

  • category - categorize weighted ads. If empty, the banner will be placed in category default.

  • weight - banner weight. Must be an integer equal or greater than 1 to consider the banner for display. 0 or blank will ignore the banner when weighted ads are built.

  • rotate - must contain a value if weighted banners are not used:

    • Empty value will prevent the banner from being displayed.

    • Literal value of 0 (zero) will cause the entire content of the banner field to be displayed when the category is matched.

    • Non-zero value will cause the banner field to be split into segments using the specified or default delimiter. For each segment, the banner will then rotate in sequence (for each website visitor separately). Obviously, the first banner in the sequence is more likely to be displayed than the last.

  • banner - the field containing actual banner content. Multiple banner texts (if this feature is used) need to be separated using the specified or default delimiter.

As usual, databases first need to be registered for use with the catalog, and so is the case with the above text database. The following modification to your catalog.cfg is needed (and a catalog reconfiguration afterwards — see ):

Database   banner   banner.txt   TAB
Database   banner   NUMERIC      weight

Now that we have everything in place, we can start experimenting with the banner tag.

To display weighted banners from a specific category (say, tech), use [banner weighted=1 category=tech]. The tag would look for all banners where the weight field is positive integer and the category matches "tech", and build them in the tmp/Banners/tech/ directory.

To display weighted banners regardless of the category, use [banner weighted=1]. The tag would look for all banners where the weight field is positive integer, and build them in the tmp/Banners/ directory. Note that the total sum of weights in our sample database is 20, so each weight point is worth 100/20 = 5%. Therefore, the banners with the weight of 7 would be displayed 35% of the time each.

To display categorized, non-rotated banners from category tech, use [banner category=tech], which is equal in effect to [data table=banner field=banner key=tech foreign=category]. Note that the value of the rotate field must be 0.

To display categorized, rotated banners (from category tech), use [banner category=tech], which is equal in effect to [data table=banner col=banner key=tech]. Note that the value of the rotate field must be 1.

To display multilevel-categorized banners (from say, category tech:hw:monitors), use [banner category=tech:hw:monitors]. The colon will help Interchange select the most specific banner available, so the difference in respect to single-level categories is the fallback mechanism; if no banners are found in the tech:hw:monitors category, the banner will try a fallback to tech:hw, and finally to tech.

For futher, supplementary reading on banners ads and standard banner sizes, check out the ad glossary entry.

1.2. Databases

Counting rows or number of items in a table

If you are using an SQL database, you can retrieve the number of rows in a database using an SQL query:

Rows: [perl products]($Db{products}->query("select count(*) from products"))[0][0][0];[/perl]

If you do not have an SQL database or want a solution that works with Interchange regardless of the underlying database used, here's a slower but generic tag from the Minivend days:

# [db-count table]
#
# This tag returns the number of records in a database table,
# 'products' by default.

UserTag  db-count  Order table
UserTag  db-count  PosNumber 1
UserTag  db-count  Routine <<EOF
sub {
  my ($db) = @_;

  $db = 'products' unless $db;

  my $ref = Vend::Data::database_exists_ref($db)
    or return "Bad table $db";

  $ref = $ref->ref();

  my $count;
  while ($ref->each_record()) { $count++ }
  return $count;
}
EOF

Thanks Ed LaFrance for providing this tip in Jan 2001.

Finding inconsistencies in Interchange table definition files

Interchange can automatically create SQL tables with proper fields and field types, once table structures has been defined in Interchange using a series of Database directives.

Structure definitions are usually kept in directory dbconf/DATABASE_NAME/, and maintained separately for each database type, which requires a great deal of manual synchronization between the files.

Here's a script that can help you find inconsistencies between database definition files.

#!/usr/bin/perl

#
# This script needs to be run from the dbconf/ directory in order to
# find the relative filenames
#

@dbtypes = qw{mysql pgsql oracle sqlite};

%dbsuff = ( mysql => mysql,
            pgsql => pgsql,
            oracle => ora,
            sqlite => lite, );
$txt = '.txt';
foreach $gtable  (@gtables = grep { s/\.txt//g } `(cd ../products/; ls *.txt)`) {
   $gtable =~ s/\n//smg;
   $all_tables{$gtable} = 1;
   $fn = $gtable . '.txt';

   $flds = `(cd ../products/; head -1 $fn)`;
   my @flds =  split /\t/, $flds;
   foreach $fld (@flds) {
      $seen{$gtable}{$fld} =1; 
   }
}


foreach $dbtype (@dbtypes) {
  # print "\n\ndbtype: $dbtype dbsuff: $dbsuff{$dbtype}\n";
  @tables = grep { s/\.$dbsuff{$dbtype}//g } `(cd $dbtype; ls *.$dbsuff{$dbtype})`;
  @tables = grep { s/\n//smg } @tables;
  foreach $table ( @tables) {
     my $M = qq{$dbtype/$table.$dbsuff{$dbtype}};
     $all_tables{$table}=1;
     # print "$M\n";
     @fields = `/bin/grep -i COLUMN_DEF $M | /bin/cut -d'"' -f2 `;
     # here I need to build a hash for each db type 
     
     foreach $field (@fields) { 
         my ($k, $v) = split (/=/, $field);
         $v =~ s/\n//smg;
         $table{$dbtype}->{$table}->{$k}= $v;
     }
  }
}
print "<html><body bgcolor=\"#ffffff\">\n";
foreach my $table (sort keys %all_tables) {
   print "<table border=\"1\" width=\"100%\">\n";
   print "<tr><td colspan=\"5\" align=\"center\"1>$table</td></tr>\n";
   print "<tr><td>&nbsp;</td>";
   foreach (@dbtypes) {
       print "<td>$_</td>";
   }
   print "</tr>";
   foreach my $field (sort keys %{$seen{$table}}) {
       print "<tr><td>$field</td>";
       foreach $dbtype  ( @dbtypes) {
          print "<td>$table{$dbtype}->{$table}->{$field}</td>";
       }
       print "</tr>\n";
   }
   print "</table>\n<br><br>\n";
}
print "</body></html>\n";

Thanks Paul Vinciguerra for providing this tip in October 2003.

Modifying text source files and applying changes

With DBM-based databases, changes to the text source files will be detected on next client access to the specific database; Interchange will then re-read the text source file and override the previous contents of the DBM database. (In case you want to disable this automatic behavior, see NoImport).

With SQL databases, the import from text source files happens only once (on database creation and initial fill). To manually trigger the re-import for a specific database, delete file DATABASE_NAME.sql from the catalog's ProductDir and reconfigure the catalog.

When SQL databases are modified through the user interface, the updates are not automatically propagated to the text source files. So when text files are updated manually and re-imported into SQL, data that was only present in SQL will be lost.

If you plan on modifying text source files even after initial import into SQL, make sure to sync tables first (using export) before making changes and re-importing.

1.3. Demo catalogs

1.3.1. "Tutorial"

Providing "quantity" field on the basket page

The page we're starting with (the one from Interchange Guides: the Catalog Building Tutorial) displays a table of the products placed in your shopping cart. The Quantity field is displayed but the only way to increase the number is to click the appropriate Order Now button multiple times, and reducing the quantity is not supported.

To remind ourselves, let's see our initial page. (This is the one used in the Interchange Guides: the Catalog Building Tutorial).

__TOP__
__LEFT__

<h2>This is your shopping cart!</h2>

<table cellpadding="5">

<tr>
  <th>Qty.</th>
  <th>Description</th>
  <th>Cost</th>
  <th>Subtotal</th>
</tr>

[item-list]
<tr>
  <td align="right">[item-quantity]</td>
  <td>[item-field description]</td>
  <td align="right">[item-price]</td>
  <td align="right">[item-subtotal]</td>
</tr>
[/item-list]

<tr><td colspan="4"></td></tr>

<tr>
  <td colspan="3" align="right"><strong>Total:</strong></td>
  <td align="right">[subtotal]</td>
</tr>

</table>

<hr>

<p>
[page checkout]Purchase now</a><br>
[page index]Return to shopping</a>
</p>

__BOTTOM__

What we need to do is:

  • Create an HTML form which is neccesary to submit any client information back to Interchange:
    <form method='post' action='[process]'>

  • Replace the Quantity label ([item-quantity]) with an HTML text field where quantity can be edited:
    <input type='text' size='2' name='[quantity-name]' value='[item-quantity]'/>

    What we have introduced here is the quantity-name tag. Interchange will expand it to the appropriate field name for each item in the cart (quantity0, quantity1, quantity2, ...). This all happens automatically and you have nothing to worry about.

  • Provide the submit button that triggers an action and sends information back to Interchange:

    [button text='Recalculate']
    mv_todo=refresh
    [/button]
    
    

  • Close the HTML form with </form>.

Here's a copy of the finished pages/ord/basket.html.

__TOP__
__LEFT__

<h2>This is your shopping cart!</h2>

<form method='post' action='[process]'>
[form-session-id]

<table cellpadding="5">

<tr>
  <th>Qty.</th>
  <th>Description</th>
  <th>Cost</th>
  <th>Subtotal</th>
</tr>

[item-list]
<tr>
  <td align="right"><input type='text' size='2' name='[quantity-name]' value='[item-quantity]'/></td>
  <td>[item-field description]</td>
  <td align="right">[item-price]</td>
  <td align="right">[item-subtotal]</td>
</tr>
[/item-list]

<tr><td colspan="4"></td></tr>  

<tr>
  <td colspan="3" align="right"><strong>Total:</strong></td>
  <td align="right">[subtotal]</td>
</tr>

<tr>
  <td colspan="4" align="right">
  [button text='Recalculate']
    mv_todo=refresh
  [/button]
  </td>
</tr>

</table>

</form>

<hr>

<p>
[page checkout]Purchase now</a><br>
[page index]Return to shopping</a>
</p>

__BOTTOM__

Test the ord/basket page in your browser. Try adjusting the quantity and pressing Recalculate. Note that setting Quantity to 0 effectively removes the item from your shopping cart.

1.3.2. "Standard"

Overriding standard Admin UI pages with per-catalog custom versions

At any time, you can copy pages from lib/UI/pages/admin/ into the pages/admin/ directory under your catalog tree to make custom versions that override the distribution ones.

You only need to copy the pages you want to override; the rest are automatically picked up from the usual, default location.

1.4. E-mail

Checking for syntactical validity of e-mail addresses

Interchange has two e-mail checks, email and <check>email_only</check>, which use a relatively simple regex in checking the syntactical validity of an e-mail address. For a way to run those profile checks on a value, see run-profile.

However, if you need as accurate and bullet-proof e-mail address matching as possible, you can use code from Chapter 7 of the Jeffrey Friedl's 1997 book "Mastering Regular Expressions", published by O'Reilly:

# Some things for avoiding backslashitis later on.
$esc        = '\\\\';               $Period      = '\.';
$space      = '\040';               $tab         = '\t';
$OpenBR     = '\[';                 $CloseBR     = '\]';
$OpenParen  = '\(';                 $CloseParen  = '\)';
$NonASCII   = '\x80-\xff';          $ctrl        = '\000-\037';
$CRlist     = '\n\015';  # note: this should really be only \015.

# Items 19, 20, 21
$qtext = qq/[^$esc$NonASCII$CRlist\"]/;               # for within "..."
$dtext = qq/[^$esc$NonASCII$CRlist$OpenBR$CloseBR]/;  # for within [...]
$quoted_pair = qq< $esc [^$NonASCII] >; # an escaped character

##############################################################################
# Items 22 and 23, comment.
# Impossible to do properly with a regex, I make do by allowing at most one level of nesting.
$ctext   = qq< [^$esc$NonASCII$CRlist()] >;

# $Cnested matches one non-nested comment.
# It is unrolled, with normal of $ctext, special of $quoted_pair.
$Cnested = qq<
   $OpenParen                            #  (
      $ctext*                            #     normal*
      (?: $quoted_pair $ctext* )*        #     (special normal*)*
   $CloseParen                           #                       )
>;

# $comment allows one level of nested parentheses
# It is unrolled, with normal of $ctext, special of ($quoted_pair|$Cnested)
$comment = qq<
   $OpenParen                              #  (
       $ctext*                             #     normal*
       (?:                                 #       (
          (?: $quoted_pair | $Cnested )    #         special
           $ctext*                         #         normal*
       )*                                  #            )*
   $CloseParen                             #                )
>;

##############################################################################

# $X is optional whitespace/comments.
$X = qq<
   [$space$tab]*                    # Nab whitespace.
   (?: $comment [$space$tab]* )*    # If comment found, allow more spaces.
>;



# Item 10: atom
$atom_char   = qq/[^($space)<>\@,;:\".$esc$OpenBR$CloseBR$ctrl$NonASCII]/;
$atom = qq<
  $atom_char+    # some number of atom characters...
  (?!$atom_char) # ..not followed by something that could be part of an atom
>;

# Item 11: doublequoted string, unrolled.
$quoted_str = qq<
    \"                                     # "
       $qtext *                            #   normal
       (?: $quoted_pair $qtext * )*        #   ( special normal* )*
    \"                                     #        "
>;

# Item 7: word is an atom or quoted string
$word = qq<
    (?:
       $atom                 # Atom
       |                       #  or
       $quoted_str           # Quoted string
     )
>;

# Item 12: domain-ref is just an atom
$domain_ref  = $atom;

# Item 13: domain-literal is like a quoted string, but [...] instead of  "..."
$domain_lit  = qq<
    $OpenBR                            # [
    (?: $dtext | $quoted_pair )*     #    stuff
    $CloseBR                           #           ]
>;

# Item 9: sub-domain is a domain-ref or domain-literal
$sub_domain  = qq<
  (?:
    $domain_ref
    |
    $domain_lit
   )
   $X # optional trailing comments
>;

# Item 6: domain is a list of subdomains separated by dots.
$domain = qq<
     $sub_domain
     (?:
        $Period $X $sub_domain
     )*
>;

# Item 8: a route. A bunch of "@ $domain" separated by commas, followed by a colon.
$route = qq<
    \@ $X $domain
    (?: , $X \@ $X $domain )*  # additional domains
    :
    $X # optional trailing comments
>;

# Item 6: local-part is a bunch of $word separated by periods
$local_part = qq<
    $word $X
    (?:
        $Period $X $word $X # additional words
    )*
>;

# Item 2: addr-spec is local@domain
$addr_spec  = qq<
  $local_part \@ $X $domain
>;

# Item 4: route-addr is <route? addr-spec>
$route_addr = qq[
    < $X                 # <
       (?: $route )?     #       optional route
       $addr_spec        #       address spec
    >                    #                 >
];


# Item 3: phrase........
$phrase_ctrl = '\000-\010\012-\037'; # like ctrl, but without tab

# Like atom-char, but without listing space, and uses phrase_ctrl.
# Since the class is negated, this matches the same as atom-char plus space and tab
$phrase_char =
   qq/[^()<>\@,;:\".$esc$OpenBR$CloseBR$NonASCII$phrase_ctrl]/;

# We've worked it so that $word, $comment, and $quoted_str to not consume trailing $X
# because we take care of it manually.
$phrase = qq<
   $word                        # leading word
   $phrase_char *               # "normal" atoms and/or spaces
   (?:
      (?: $comment | $quoted_str ) # "special" comment or quoted string
      $phrase_char *            #  more "normal"
   )*
>;

## Item #1: mailbox is an addr_spec or a phrase/route_addr
$mailbox = qq<
    $X                                  # optional leading comment
    (?:
            $addr_spec                  # address
            |                             #  or
            $phrase  $route_addr      # name and address
     )
>;


###########################################################################
#
# The regex used in matching is built in $mailbox.
#
# Here's a little snippet to test it.
# Addresses given on the commandline are described.
#

my $error = 0;
my $valid;
foreach $address (@ARGV) {
    $valid = $address =~ m/^$mailbox$/xo;
    printf "`$address' is syntactically %s.\n", $valid ? "valid" : "invalid";
    $error = 1 if not $valid;
}
exit $error;

The program can be saved to a file and ran for testing e-mail addresses validity on the command line.

For use with Interchange, you don't need the whole program — simply print the final regex generated in $mailbox and then copy it to the location where you need it.

Optimizing e-mail delivery with a custom Sendmail routine

It was observed in 2004 that about once in every ten times, the etc/mail_receipt takes a few extra seconds before the receipt is mailed out.

The delay occured because in the code used, the SendMailProgram ran in foreground so the ordering process had to wait for Sendmail to finish sending e-mail. The critical section was reduced to:


[email to="[scratch to_email], __MAIL_RECEIPT_CC__"
       subject="__COMPANY__ Order #[value mv_order_number]: [scratch subject_end]"
       from=|"__COMPANY__ Order Confirmation" <orders at company.com>| ]

... email contents ...

[/email]


To eliminate this "delay", you could use a custom script that accepts the message on standard input (STDIN), returns control back to your code immediately and calls Sendmail in the background.

Save the script with the following contents to /usr/local/bin/sendmail-bg:

#!/usr/bin/perl

#use strict;
#use warnings;
use File::Temp;

my $basedir = '/tmp/sendmail';
my $sendmail = '/usr/sbin/sendmail -t';
umask 2;

mkdir $basedir unless -d $basedir;
my $tmp = File::Temp->new( DIR => $basedir );
my $tmpnam = $tmp->filename;

open OUT, "> $tmpnam" or die "Cannot create $tmpnam: $!\n";
my $cmdline = join " ", $sendmail, '<', $tmpnam, '&';
while(<>) { print OUT $_; }
close OUT;

system($cmdline);

if($?) { die "Failed to fork sendmail: $!\n" }

And set SendMailProgram directive in catalog.cfg:

SendMailProgram /usr/local/bin/sendmail-bg

1.5. Forms and form submissions

Defining mv_metadata width and height for HTML textarea

In mv_metadata, you cannot use the usual width and height fields to set HTML <textarea> size.

Use the following instead (for a 5x50 characters box):

fieldname:
  widget: textarea_5_50
  height:
  width:

Testing for errors in form submissions

To quickly test whether any errors were set in the last form submission and display them, use:

[if errors]
  [error all=1 show_error=1 joiner="<br>"]
[/if]

The code that works with Minivend 4.02 is as follows:

[if type=explicit compare="[error all=1 keep=1]"]
  ......
[/if]

1.6. Ordering

Forcibly adjusting quantities of ordered items

Your stock may be limited, or you might want to force order quantities for other reasons.

There are a number of ways to do this. Here is one; the code should be placed at the top of the basket and checkout pages. It operates on the cart "main" and prevents people from ordering more than what is available according to quantity field in the inventory:

[perl tables=inventory]
  my $cart = $Carts->{main};
  my $item;
  foreach $item (@$cart) {
    my $on_hand = tag_data('inventory', 'quantity', $item->{code});
    next if $on_hand >= $item->{quantity};
    $item->{quantity} = $on_hand;
    $item->{q_message} = "Order quantity adjusted to fit stock.";
  }
[/perl]

You can place [item-modifier q_message] somewhere in the item line to notify the customer of adjusted quantity.

1.7. Searches

Avoiding search strings in HREF= specifications

You should never embed searches in a HTML HREF= specification, because then you have to worry about escaping and formatting and things become fragile.

So you should never do something like:

[area href="scan/lf=category/ls=%Hot Dogs"]

The proper way to go about it instead, is to use any of the below two methods:

[area search="
  lf=category
  ls=%Hot Dogs
"]

or

[area href=scan arg="
  lf=category
  ls=%Hot Dogs
"]

Creating custom search routines for use in mv_column_op

It is possible to define your own search functions that can be specified in mv_column_op instead of the usual, built-in operations.

See CodeDef reference page for examples of creating SearchOps.

Once you've created the SearchOp, you can use something like the following ad-hoc specification to test it (this example works with the SearchOp example from CodeDef):

[loop search="
  se=rubber hammer
  sf=description
  fi=products
  st=db
  co=yes
  rf=*
  op=find_hammer
"]

[loop-code] [loop-param description]<br>

[/loop]

Supporting AND, OR, and other advanced search specifications

You can install Text::Query Perl module to enable advanced query parsing for search specifications.

To use Text::Query, set mv_column_op to "aq" for advanced parsing or "tq" for simple parsing.

To learn about the features "aq" and "tq" give you, see Text::Query::ParseAdvanced(3pm) and Text::Query::ParseSimple(3pm) manual pages.

You can also make the use of Text::Query conditional, based on whether Text::Query has been installed on the server:

<input name="mv_column_op" type="hidden" value="[if module-version Text::Query]aq[else]rm[/else][/if]">

Text::Query parsing honors variables mv_case (-case option), mv_all_chars (-regexp option) mv_substring_match (-whole option) and mv_exact_match (-litspace option).

Ad-hoc examples to test search specifications might look like:

[loop search="
            se=hammer -framing
            sf=description
            fi=products
            st=db
            co=yes
            rf=*
            op=tq
        "]

[loop-code] [loop-param description]<br>

[/loop]


[loop search="
            se=hammer NEAR framing
            sf=description
            fi=products
            st=db
            co=yes
            rf=*
            op=aq
        "]

[loop-code] [loop-param description]<br>

[/loop]

If you have a custom search method and want to manually support AND and OR in search specifications while using the usual "eq" and "rm" column operations, use something like the following to pre-process the search specification:

[calc]
  my $text_qualification = 'category = Hammers and price < 15';


  $CGI->{text_qualification} = <<EOF;
fi=products
st=db
co=1
  EOF
  
  my @entries = split /\s+(and|or)\s+/i,  $text_qualification;

  my $or;
  for(@entries) {
    if(/^or$/i) {
      $or = 1;
      $CGI->{text_qualification} .= "os=1\n";
      next;
    }   
    elsif(/^and$/i) {
      $or = 0;
      $CGI->{text_qualification} .= "os=0\n";
      next;
    }   
    my ($f, $op, $s) = split /\s*([<=!>\^]+)\s*/, $_, 2;
    $op = "eq" if $op eq "==";
    $op = "rm" if $op eq "=";
    if($op eq '^') {
      $op = 'rm';
      $CGI->{text_qualification} .= "bs=1\nsu=1\n";
    }   
    else {
      $CGI->{text_qualification} .= "bs=0\nsu=0\n";
    }   
    $CGI->{text_qualification} .= "se=$s\nsf=$f\nop=$op\n";
    if($op =~ /[<>]/ and $s =~ /^[\d.]+$/) {
      $CGI->{text_qualification} .= "nu=1\n";
    }   
    else {
      $CGI->{text_qualification} .= "nu=0\n";
    }   
  }   
  [/calc]

Then use the following to run the search and display search results:

<pre>

Search for: [cgi text_qualification]

Results:

[loop search="[cgi text_qualification]"]
  [loop-code] [loop-description]
[/loop]

</pre>

Thanks Mike Heins for providing this tip in January 2001.

1.8. Server Process

Adjusting timezone

You can adjust time globally for an Interchange installation by setting the $ENV{TZ} variable on many systems. Set TZ in your environment by one of:

## bash/ksh/sh
TZ=PST7PDT; export TZ

## csh/tcsh
setenv TZ PST7PDT

interchange -restart

Dumping complete configuration ($Vend::Cfg)

To dump the complete config structure at will, run:

[uneval ref=`$Config`]

Also, at catalog startup, a file named etc/CATNAME.structure will be created relative to the catalog root directory. A complete configuration dump will be saved there.

Testing configuration, starting, reconfiguring, stopping

Knowing how to manage the Interchange daemon is one of the very basic administration tasks.

[Note]Note

Since Interchange does not generally like being run with admin privileges (user root), you will most probably have to invoke the commands listed below under the Interchange username. To do so, you will either log in as Interchange user, or as sufficiently-privileged user invoke su -c 'COMMAND' INTERCHANGE_USER, or sudo -u INTERCHANGE_USER COMMAND.

Configuration is tested without interrupting the running processes by invoking:

  • interchange -test

Interchange is (re)started by one of:

  • /etc/init.d/interchange restart (Debian GNU)

  • /etc/rc.d/init.d/interchange restart (Red Hat)

  • interchange -r (tarball)

Specific catalogs are reconfigured by one of:

  • Running interchange -reconfig CATNAME

  • Using the tag reconfig on an Interchange page

  • Touching a file named after the catalog within the etc/reconfig/ directory in ICROOT.

Interchange is stopped by one of:

  • /etc/init.d/interchange stop (Debian GNU)

  • /etc/rc.d/init.d/interchange stop (Red Hat)

  • interchange -stop (tarball)

[Note]Note

Because processes that are waiting for selection can block signals on some operating systems, it is possible that you have to wait for HouseKeeping seconds before the server really closes down. To terminate the server with prejudice (in the event it will not stop), use interchange -kill.

1.9. Sessions

Expiring

See the expire glossary entry.

Reading and writing user sessions

User sessions are kept in a particular type of database as controlled by SessionType configuration directive.

Depending on the purpose, sessions can be dumped using tags dump or dump_session.

If you are going to do something specific to a session, such as process it later with an external Perl program, you could first save it as text representation of the Perl hash to an explicitly-named file tmp/SESSIONID.save, by invoking something along the lines of:

my $sess_string = $Tag->uneval( { ref => $Session } );
$Tag->writefile("tmp/$Session->{id}.save", $sess_string);

Then, in your retrieve routine (which must be a global usertag if you will be doing it from Interchange), do something like:

my $safe = new Safe;
my $sess_string = $Tag->file("tmp/$id_to_retrieve.save");
my $session_ref = $safe->reval($sess_string);

Storing sessions in MySQL

Interchange can use databases to store user sessions. The corresponding catalog.cfg definition for MySQL-based sessions is:

Message Starting MySQL-based sessions setup...

SessionType DBI

Database  session  session.txt dbi:mysql:sessionfiles:localhost
Database  session  USER         username
Database  session  PASS         password
Database  session  KEY          code
Database  session  COLUMN_DEF   "code=varchar(64) NOT NULL PRIMARY KEY"
Database  session  COLUMN_DEF   "session=blob"
Database  session  COLUMN_DEF   "sessionlock=VARCHAR(64) DEFAULT ''"
Database  session  COLUMN_DEF   "last_accessed=TIMESTAMP(14)"

SessionDB   session

Message ...Done.

Thanks Dan Browning for providing this tip in January 2002.

1.10. Tax / VAT

Implementing basic, country- and category-based tax/VAT scheme

Many european countries use country or category-dependent tax rates.

The basic idea is to write a usertag that will return the tax amount. The way usertag reaches the final amount can then be modified or improved without having to make other changes to the configuration. Let's create the basic tag, include it in interchange.cfg or catalog.cfg, and call it from the salestax table.

The basic tag that returns the sum of taxes for all individual products in user's basket is as follows:

UserTag  vat-calc  Order  table field
UserTag  vat-calc  addAttr
UserTag  vat-calc  Routine <<EOR
sub {
  my ($table, $field, $opt) = @_;
  
  my $error = sub {
    my $msg = shift;
    Log($msg);
    return undef;
  };

  my $tax = 0;
  foreach my $item (@$Vend::Items) {
    my $taxrate = tag_data($table, $field, $item->{code});
    $tax += ($taxrate * $item->{quantity});
  }
  return $tax;
}
EOR

If you add the above code in catalog.cfg and not in interchange.cfg, you will have to make sure you "open" the products database before this code runs, by using [perl products][/perl] if nothing else.

To use built-in tax support in Interchange and to call our new vat-calc tag, create salestax.asc as follows:

default [vat-calc products tax]
UK  [vat-calc products tax]
FR  [vat-calc products tax]
US  0

For country-based tax selection, you could modify salestax.asc to invoke vat-calc with different database or field options depending on country, and enable country lookup itself by using the following in catalog.cfg:

SalesTax country

For category-dependent tax rates, you would modify the vat-calc tag to take $item->{category} into account and enable the "category" modifier as follows:

AutoModifier products:category

Implementing user-based tax/VAT scheme

To set up tax-exempt users, expand your userdb database to add field tax_exempt.

Have your SalesTax defined as usual, as if no users will be tax-exempt. Then, make the following changes to catalog.cfg to unset SalesTax for tax exempt users:

UserDB default scratch tax_exempt

AutoLoad <<EOL
[calc]
if ($Scratch->{tax_exempt}) {
	$Config->{SalesTax} = ' ';
}
return;
[/calc]
EOL

Thanks Greg Hanson for providing this tip in March 2001.

1.11. User database

Searching through UserDB

While you can normally access data in UserDB, searching it using Interchange search functions is prevented by default (using NoSearch).

To temporarily allow searching and perform a search for an email address, use something like the following:

[calcn]
	$Config->{NoSearch} =~ s/userdb//;
	return;
[/calcn]

[loop search="
 st=db
    fi=userdb
    ml=1
    sf=email
    se=[value email]
    rf=username, password
    "
]
	[seti userdb_username][loop-field username][/seti]
	[seti userdb_password][loop-field password][/seti]
[/loop]

Thanks Hans-Joachim Leidinger for providing this tip in December 2000.

2. HTML

Validating (X)HTML markup

You can validate pages at W3C using CSS Validator or Markup Validator.

Interchange outputs a lot of HTML markup. The markup is a bit inconsistent in style, casing (uppercase/lowercase) and argument quoting, but an effort has been underway to standardize on all-lowercase, properly-quoted markup only.

Lowercasing and quoting arguments does not harm compatiblity with old web browsers, and is a nice step towards strict XHTML compliance.

For XHTML compliance, non-container markup tags should be closed with /> instead of just >. Enable the XHTML configuration directive which will adjust the output markup on internally-generated non-container HTML tags.

There's the $Vend::XTrailer variable which you can also use in your own code to automatically output literal '/' or nothing before the closing of a non-container tag, according to XHTML.

If you patch Interchange source to produce better HTML markup, please send your patches to our users list, interchange-users.

3. Web servers

3.1. Apache

Logging to per-vhost logfiles

Add the following to your Apache vhost configuration:

ErrorLog /var/log/apache/myhost.mydomain.local.error.log

CustomLog /var/log/apache/myhost.mydomain.local.log \
  "%h %l %u %t \"%r\" %<s %b \"%{Referer}i\" \"%{User-Agent}i\""

<IfModule mod_ssl.c>
  CustomLog /var/log/apache/myhost.mydomain.local.ssl.log \
    "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
</IfModule>

Using rewrite rules to avoid /cgi-bin/ in URLs

The Apache config file (or a corresponding vhost file) needs something like the following:

RewriteEngine   on

RewriteRule     ^$      /cgi-bin/shop/  [L,NS,PT]
RewriteRule     ^/$      /cgi-bin/shop/  [L,NS,PT]
RewriteRule     ^/index.html$      /cgi-bin/shop/  [L,NS,PT]

RewriteCond     /home/shop/www/html/%{REQUEST_FILENAME} -f      [OR]
RewriteCond     /home/shop/www/html/%{REQUEST_FILENAME} -d
RewriteRule     ^(.+)$  -  [L,NS]

RewriteCond     /%{REQUEST_FILENAME}    !^.*/cgi-bin/shop.*$
RewriteRule     ^(.+)$  /cgi-bin/shop/$1  [L,NS,PT]

Thanks Frederic Steinfels for providing this tip in March 2003.

4. Databases

4.1. PostgreSQL

4.2. MySQL

Fixing constantly-failing SQL statements

One possible cause for the failing MySQL statements is the too-low timeout value. Try increasing the timeout to say, 50 minutes, by inserting the following in my.cnf:

set-variable = wait_timeout=3000

Thanks Frederic Steinfels for providing this tip in March 2004.

5. Unix

Monitoring log files

To monitor all relevant log files at once, use something like:

cd /var/log; tail -f {apache,postgresql}/*log mysql.{err,log} interchange/error.log /PATH/TO/catalogs/*/var/log/*.log

If you want selective results, colorized output, or output to a root window in your X session, see grep, glark, ccze, colorize or root-tail.

Add the following to your global Interchange configuration:

Variable DEBUG 1
DebugFile /var/log/interchange/debug.log
ErrorFile /var/log/interchange/error.log

Add the following to your Interchange catalog configuration:

ErrorFile var/log/error.log
TrackFile var/log/track.log

6. Perl

Making Perl hash modifiable without overriding original values

Sometimes you want to make a data structure modifiable, without having the changes permanently override original values. To do so, we can use Perl's Tie::ShadowHash module.

This is an old trick, and generally not needed in Interchange, as it already supports modification of configuration directives for the duration of the page only (which is usually done in an Autoload routine).

However, you might find the recipe useful. Here's the usertag:

UserTag modifiable Order thing
UserTag modifiable Routine <<EOR
require Tie::ShadowHash;
sub {
  my $thing = shift || 'Variable';
  my $ref = $Vend::Cfg->{$thing};
  return undef if ref($ref) ne 'HASH';

  my %hash;
  tie %hash, 'Tie::ShadowHash', $ref;

  my $new = \%hash;
  $Vend::Cfg->{$thing} = $new;

  if($thing eq 'Variable') {
    $::Variable = $Vend::Interpolate::Variable = $new;
  }

  return "$thing set to modifiable";
}
EOR

To test it, establish initial value of a variable in catalog.cfg:

Variable FOO bar

And run the following code:

FOO=__FOO__<br>
FOO=[var FOO]<br>
[modifiable Variable]<br>
Set...[calc] $Variable->{FOO} = 'hosed'; [/calc]<br>
FOO=[var FOO]

The output should be the same on every page reload, meaning that the new value overrides the initial one for the duration of the page only.

DocBook!Interchange!