![[BACK]](/cvsweb/icons/back.gif){kind=icon}
Up to [Local Repository] / interchange / lib / Vend
Request diff between arbitrary revisions
Keyword substitution: kv
Default branch: MAIN
* mv_force_coordinate as discussed on http://www.icdevgroup.org/pipermail/interchange-users/2008-July/049222.html
* Updated copyright headers to 2007 (closes bug #102) * Added GPL and copyright headers to a few files that were missing them.
New Free Software Foundation Address in headers of various files
New Free Software Foundation Address in headers of various files
log errors for custom search module initialization errors, standard message for users
Big copyright and version number update to prepare for 5.3.2 release.
* We had been running without warnings so long, we had gotten a bit sloppy
in our code in some places.
Because we do lots of uninitialized hash member stuff with arrays
of fields and $opt, it is basically unreasonable for us to run with
the "use of uninitialized variable" warnings. Because we do a fair
amount of numerical comparisons on empty string values, it seemed
unreasonable to run with "string value in numeric gt" enabled.
Also, because Safe, HTML::Entities, and some other modules have
these checks enabled, page code is bound to have warnings issued
no matter what you do.
It seems better to run with some warnings enabled so that we can
catch bad code practice.
So I have made the code run without warnings prior to and after
interpolate_html(), and turned off warnings by default in the
page code.
If it is desirable to turn on warnings for pages, you can use
[pragma perl_warnings_in_page] and see warnings for that page
(or by default if you put in catalog.cfg).
* In general these changes are:
no warnings qw(uninitialized numeric);
in most pages.
* Add $::Pragma->{perl_warnings_in_page}.
* Remove defined checks when simple set tests will do.
* Streamline some code.
* Add to scan order so that permanent more will work for searches as well as loops/queries.
* Add "permanent more" feature so that you can create pageable searches that are shared between people and are cacheable by search engines. * To activate, put in search URL/form: pm=1 or mv_more_permanent=1 * To activate in [query ...], use pm=1. * Changes hashing to two levels so that it will handle larger sets of searches. * Depends on consistent spacing of queries, caches query for mv_cache_key that determines location. The change of a single bit of whitespace will change the key! * When your data changes, a saved search could be wrong when referenced from a bookmark/saved link and no initializing search has been done. * WARNING: Could generate very large file sets, watch your inodes!
* Don't save more matches in [loop ...] unless more=1 is set. * Fix typo causing mv_no_more to be ignored in certain circumstances.
A few Search enhancements.
* Add ability to save search paging files in the SessionDB when using
DBI sessions. This obviates the need for an NFS-shared tmp/ directory
when sharing accross multiple servers.
Requires setting in catalog.cfg:
MoreDB Yes
Setting this to "Yes" if not using DBI sessions will cause errors.
* Allow passing of mv_no_more (nm) parameter which states that no more
should be done. This is particularly useful for searches like are done
in the "random" component, where we have a random=N with a ml=foo.
That always means a more save, which adds overhead.
Causes save_more to return true without saving.
TODO: Examine automatically adding to [loop ...] lists that don't have
more=1 passed as an option.
* Add ability to save search paging files that are non-session-specific
in a query tag. This allows:
[timed-build force=1 file="timed/[cgi letter][cgi mv_first_match]"]
[query
list=1
mv_more_id=nsession
more=1
ml=10
sql="SELECT * FROM products where description like '[cgi letter]%'"
]
....
[/query]
[/timed-build]
A completely duplicatable paged list is saved with a minimum of overhead;
each paging is done only once and the [more] will use a common source.
TODO: make sure this works with the prefix paging stuff...
* Improve SQL_Parser to recognize individual "distinct" fields and the header value of "column as 'Title'". * Reserve mv_header_fields in search spec, not handled in search specs yet.
* Set up error reporting to be able to catch database errors
and display in session, catalog error.log, or global error.log
1. Logging levels are on a per-table basis, with
defaults that can be set with DatabaseDefault:
DatabaseDefault LOG_ERROR_CATALOG 1
DatabaseDefault LOG_ERROR_SESSION 1
DatabaseDefault LOG_ERROR_GLOBAL 0
DatabaseDefault DIE_ERROR 0
2. Log errors to the catalog error.log by default.
Database inventory LOG_ERROR_CATALOG 0|1*
3. Log errors to the session always if an admin, and
controlled by configuration if not.
Database inventory LOG_ERROR_SESSION 0|1*
This has the effect of giving a big red error message when such
an event as failing to create a record occured. In most cases,
you would be able to use the <-Back button and fix the error
and resubmit.
The error tag is "table foo", where foo is the table.
4. Die at the page level (500 error) only if that is explicit
request in config for that table:
Database inventory DIE_ERROR 0*|1
5. Log errors globally only on explicit request:
Database inventory LOG_ERROR_GLOBAL 0*|1
6. LENGTH_EXCEPTION errors go into warnings if they are handled
with truncate.
* default
* Fix numeric sorting in SQL statements if a field is NUMERIC.
* Allow limits from SQL statement to flow through even if ml="" is set
and let direct_sql.html admin page honor them.
* Fix table names so that we don't have the funky .txt problems
where a SQL query would not work on a DBM database unless
the file name base matched the table name.
* Attempt to regularize error messages so that they can be
more easily translated. Now should have about 50% less
variations.
* Add Altavista-style search operator with Text::Query (CPAN) module.
Calls Text::Query::*AdvancedString with op=aq, calls
Text::Query::*SimpleString with op=tq.
Examples:
[loop search="
se=hammer -framing
sf=description
fi=products
st=db
co=yes
rf=*
op=tq
"]
[loop-code] [loop-param description]<br>
[/loop]
[loop search="
se=hammer NEAR framing
sf=description
fi=products
st=db
co=yes
rf=*
op=aq
"]
[loop-code] [loop-param description]<br>
[/loop]
Honors mv_case (-case option), mv_all_chars (-regexp option),
mv_substring_match (-whole option) and mv_exact_match
(-litspace option).
* Add ability to map in custom search routines. In interchange.cfg:
CodeDef find_hammer SearchOp find_hammer
CodeDef find_hammer Routine <<EOR
sub {
my($self, $i, $string, $opname);
#::logDebug("Calling fake SearchOp");
return sub {
#::logDebug("testing with fake SearchOp");
my $string = shift;
$string =~ /hammer/i;
};
}
EOR
Now you can do:
[loop search="
se=hammer NOT framing
sf=description
fi=products
st=db
co=yes
rf=*
op=find_hammer
"]
[loop-code] [loop-param description]<br>
[/loop]
The passed parameters are:
- The search object ($self)
- The index into coordinated search array ($i)
- The pattern to match
- The name of the op (find_hammer in this case)
Must return a sub which receives the data to match and returns
1 if it matches. DOES NOT HONOR mv_negate UNLESS you tell it to.
See Vend::Search::create_text_query for an example of how to
return a proper routine and look in search object for the
associated params.
* Fix several deficiencies in SQL parsing.
-- Recognize IS [NOT] NULL and map to a search for the
empty string.
-- Allow verbatim passing of field names for GDBM types, allowing
"select Variable from variable where Variable = ''" which
would not work before.
-- Add VERBATIM_FIELDS definition to database types which need it.
-- Add support for "select sometable as foo, othertable bar where ..."
so that queries using it can be rerouted properly.
-- Always set mv_min_string = 0, so we don't have to do anything
special for "where column = ''" and such.
* Remove other SQL::Statement dependencies and requirements.
* Add Vend::SQL_Parser module, eliminating need for SQL::Statement. * Improved tolerance for re-routing queries with table-only option. * Parses more SQL -- now can use IN and BETWEEN and translate those to IC search specs. * Handles complex parenthisized queries properly. * Reads LIMIT N and translates to mv_matchlimit. * Tests added to regression tests to check parser.
The great copyright, email address, URL, and version update.
* Add new Vend::File module with minimal functions. * Relocate following routines (and their subsidiaries) from Vend::Util: canonpath catdir catfile exists_filename file_modification_time file_name_is_absolute get_filename lockfile path readfile readfile_db set_lock_type unlockfile writefile Added stubs so that package-based calls to those routines will not break software. * Added CatalogUser directive that allows setting in interchange.cfg of allowed username that is used for access to absolute-path names. CatalogUser foundation joe CatalogUser reports jane This sets the user for allowed_file() for further read/write checks based on username. * Created allowed_file() routine and replaced all current inline checks for NoAbsolute with call to that routine. It behaves as: NoAbsolute is No: all files are accessible, always Allowed for read/write: Path name is relative with no .. Path name is absolute but in the catalog directory Path name is absolute but in a TemplateDir Allowed for read: CatalogUser set to a valid username and file is readable by that user CatalogUser set to a valid username and file is readable by a group containing that user Allowed for write: CatalogUser set to a valid username and file is writable by that user CatalogUser set to a valid username and file is writable by a group containing that user * Changed display_special_page so that special page entries with ../ will not break things. * TODO: -- Code read for open() calls. -- Code read for chmod() calls. -- Code read for unlink() calls. * Passes all regression tests; takes an order on foundation; runs UI including file navigator.
updated LINUXIA branch to 4.9 sources in order to use it as testbed again
Sweeping update of Akopia/Red Hat references, to prepare for 4.8 release with current Interchange URLs and contact information.
* Allow searching for the empty string if mv_min_string = 0. * Prevent generation of bogus limit function when SQL has already done a "select fields from table where field = 'foo'".
* Patches to make instant database handling of delimited files work. * Changed _file_security_scalar in Scan.pm to use the standard file_name_is_absolute() routine instead of homegrown check. Which exposed a rather nasty little fault in the logic there. -- If the search type is db, and the basename of the source file for the table does not match the table name, there is a big problem. The table name will be transformed by this routine, and the search will fail with a "foo non-existent table" error. I don't think this has bitten very many people, because in practice most people name the file the same as the table, but it should be corrected. Unfortunately, we don't know the search type until after this is done. My idea is that we no longer be able to find the file name for a text search based on the table name. I doubt many people were intentionally using this feature, and all you have to do is pass "products.txt" on the fi= spec to solve the problem.
* Add new "instant database", which allows you to build a database for
editing or any other use by specifying a file name.
The file name must be relative, even if NoAbsolute is not set, and
it must end in .txt or .asc. (Add .csv?)
Imported every time, of course, but only written when something
has been changed (by flagging a _Dirty bit).
[flag type=write table=Content_txt]
[data include/menus/Content.txt page 001]
[data table=include/menus/Content.txt col=page key=001 value="YES!"]
[data include/menus/Content.txt page 001]
[perl tables="Content_txt"]
my $db = $Db{Content_txt};
my $ary = $db->query('select code,page,name from Content_txt');
my $count = 0;
for(@$ary) {
push @out, uneval($_) . "\n";
$count++;
}
return join "\n", "Showed $count records in Content_txt.", @out;
[/perl]
All normal database operations seem to work.
* Fix a bug in escape_form that didn't unescape %NN parms.
* Remove commented-out code never used.
* Enable mv_small_data search control, prevents limit function.
* Add new HIDE_FIELD capability to DbSearch. It provides automatic hiding of records accessed via search (and NOT query). -When the following configuration is added: Database products HIDE_FIELD inactive -It adds automatically the qualification to every search: WHERE inactive != 1 -To use, you should have a field of char(1) or int type. Database products COLUMN_DEF "inactive=int default 0" -This has the side-effect of hiding fields with NULL in the field, so be careful. You should probably set "default 0" as shown above. -Works for DBM types too. -Does NOT work for TextSearch. -If you want to show all records, you can pass mv_no_hide=1 in the search parameters. Obviously, this makes this not a security feature.
Add option to disable matchlimit entirely, with ml=all or ml=none. Allow default of 50 to be changed in catalog variable MV_DEFAULT_MATCHLIMIT.
Implementing mv_max_matches as done in stable branch earlier today.
Implementing mv_max_matches: if > 0, the search results will spliced at its offset.
Big copyright date update. This will be the last time that Red Hat copyright notices get updated en masse, since the bulk of new code will very likely come from non-Red Hat employees.
* Add new mv_next_search, which allows filtering searches
through another search after the first one returns
some rows.
* Add search_reference() method to Search.pm
* Add st=ref to allow searching of an array of arrays with
an IC search -- I would not think it too useful but this
type of thing has been requested
* New Vend::RefSearch actually does the search
* Search is passed by setting in a scratch variable right now,
suggestions on how to better pass it are appreciated
* A small test will be posted to core list.
backported two bugfixes from development branch, which experienced enough testing IMHO: 1) introduced new Vend::Util::escape_scan_url function which escapes unsafe characters in string (I consider the = sign safe in opposite of what URI::Escape says, because it looks to ugly and is bloat in IMHO to escape it) therefore these things should work now as expected: [area href="Catalog/Food/Hot Dogs"] [area href="scan/lf=category/ls=%Hot Dogs"] 2) Vend::Util::errmsg don't call sprintf anymore if only one parameter is passed which protects against unwanted expansion
* Fix escape_mv problem that required find_search_params() to escape values. As I thought, this was a legacy of my 1996 use of period to escape characters and the continuance through the end of MV3 to support that. We no longer ever escape characters in escape_mv (a misnomer, now, I guess) and all seems to work well.
* Back out change with urldecode handling. It is indeed needed in some situations. Analysis to follow....
remove probably unneeded and mischievous expansion of % sequences
Update version number and copyright year, to prepare for the eventual day when we release version 4.8.4.
Remove last remnants of mv_raw_searchspec, as per Mike's email of some time ago: Date: Tue, 11 Sep 2001 08:23:20 -0400 From: mheins@redhat.com To: interchange-users@interchange.redhat.com Subject: Re: [ic] mv_raw_searchspec remains undefined Quoting Craig Carman (webmaster@seeshopgo.com): > Dear IC Acquaintances, > > Where and how do I learn the use of mv_raw_searchspec (SE) since > it seems to be undocumented? It shouldn't be a part of the code or docs anymore. It is meaningless. -- Red Hat, Inc., 3005 Nichols Rd., Hamilton, OH 45013 phone +1.513.523.7621 <mheins@redhat.com> For a successful technology, reality must take precedence over public relations, for Nature cannot be fooled. -- Dick Feynman
Merge from 4.9 the error logging when search is rejected because of NoSearch directive.
Clean up logic on NoSearch logging.
* Fix phantom database problem researched by Chris Ochs. The problem
was auto-vivification of a Vend::Cfg->{Database} entry, causing a
problem in PreFork mode.
This problem was plaguing a lot of people and I wouldn't have found
it without Chris's fine work.
* Fix phantom database problem researched by Chris Ochs. The problem
was auto-vivification of a Vend::Cfg->{Database} entry, causing a
problem in PreFork mode.
This problem was plaguing a lot of people and I wouldn't have found
it without Chris's fine work.
Log message if search is denied because of NoSearch directive.
Upping major version number after trunk/branch switch.
Move DEV_4_7_0 development branch to repository trunk.
* Change sql_statement error handling to always die upon parse error. This will allow the calling routines to trap errors better -- when we got a problem before we were getting a "can't use string as HASH ref" message that was inappropriate. This should have never been an entry point for outside callers, and even if it was an unexpected die was happening anyway.
Update copyright notices and email addresses. Standardize module headers and remove a few shebang lines.
Fix a problem detecting if SQL::Statement is available.
For some reason, under some circumstances, %INC gets emptied, I haven't
figured out why, but to work around the problem this patch changes
unless $INC{'SQL/Statement.pm'} to unless defined &SQL::Statement::new, which
has the same effect without depending on %INC.
This patch introduces more links based on the first letters of the search result instead of partition them by number, so it is really easy to figure out which more link one has to click to get a certain result page. I think this may be prove useful for the UI too, but I mark it as experimental code. Documentation and bugfixes are planned. Currently it is rather seperated from the rest of the code, so I don't expect it to break anything. Short example: <input type=hidden name=mv_search_field value=partnumber> <input type=hidden name=mv_sort_field value=partnumber> <input type=hidden name=mv_more_alpha value=yes> <input type=hidden name=mv_return_fields value=idf,partnumber> Comments are highly appreciated.
Fix bug in SQL::Statement (wouldn't accept a zero/blank search spec).
* Add mv_like_field and mv_like_spec to search specifications,
designed to filter SQL (only!) searches with
mv_like_field like 'mv_like_spec'.
This is a stackable field/spec set like mv_search_field
and mv_searchspec, and will eliminate any fields with empty
mv_like_spec values.
Checks the known_capability to see if UPPER_COMPARE is set
for that database, and uses (pseudo-code) "UPPER($col) like "\U$spec"
if that is the case.
* Add known_capability UPPER_COMPARE (set for Pg and Oracle to
begin with) to allow upper-case transforms for case-insensitive
compares.
Add SkipjackIC payment routine, contributed by Matthew Schick <mschick@brightredproductions.com>, and modified somewhat by me. Thanks Matthew!
Whitespace changes (diff -b -B will report nothing except below): 5 additional lines of diff (whitespace not caught by -b -B)
merge 1.7 vs. 1.8 also removed cuddled else statements
merged changes 1.7 vs 1.8
* Fixes to make mv_sql_query work properly. There was a
push_spec('nu' ...) call which was in the wrong place, making
numeric hash all out of sync.
Fix problem with nu parameter getting out of sync in mv_sql_query.
Merge 1.6 vs. 1.7.
merged changes 1.6 vs 1.7
Fix call of non-imported routine, reference from main package. (Why isn't it imported properly?)
stylistic changes
merged changes from trunk
merged changes from main trunk in
* Allow setting of default mv_searchtype to "db" if Variable
MV_SEARCH_DEFAULT_DB is set.
Added mv_more_id support.
Added multiple security fixes. Added beginnings of IPC to perl programs. Added Sub (catalog.cfg) link to [PREFIX-exec]. Added hs (head skip) option to [import ....] tag.
Changed Interchange name all over, minor exceptions being mvfaq.pod and mvrpm.pod. Changed copyright notice to Akopia.
Added mv_start_match to find first key matching specification. Like mv_first_match, except sets based on value of match and not index of match. Added more security to Session.pm, saving username safe from manipulation by user.
Initial import of changed Challenger with UI instead of Minimate.
Initial revision