[ic] [import] mangles UTF-8 characters

Stefan Hornburg (Racke) racke at linuxia.de
Tue Apr 20 08:39:02 UTC 2010


Stefan Hornburg (Racke) wrote:
> Stefan Hornburg (Racke) wrote:
>> Hello,
>>
>> we are in the progress to migrate two projects from a patched
>> 5.7.1 installation to 5.7.6.
>>
>> These are the relevant settings in catalog.cfg:
>>
>> Variable MV_HTTP_CHARSET UTF-8
>> DatabaseDefault PG_ENABLE_UTF8 1
>>
>> In etc/log_transaction this code creates the orderline entries:
>>
>> [import table=orderline type=LINE continue=NOTES]
>> ...
>> [/import]
>>
>> This fails if items carry UTF-8 characters in their name/description,
>> e.g.:
>>
>> import into orderline failed: DBD::Pg::st execute failed: ERROR:  
>> invalid byte sequence for encoding "UTF8": 0xf1612028
>>
>> Similar code using [query] instead of import works.
>>
>> This was discussed on IRC yesterday, and we found that the following
>> two code parts probably need to examined in order to fix this bug:
>>
>> Vend::Data, import_text, 303ff (writing temporary file)
>>
>> if($options->{file}) {
>>     $fn = $options->{file};
>>     Vend::File::allowed_file($fn)
>>         or die ::errmsg("No absolute file names like '%s' allowed.\n", 
>> $fn);
>> }
>> else {
>>     Vend::Util::writefile($fn, $text)
>>         or die ("Cannot write temporary import file $fn: $!\n");
>> }
>>
>> Vend::Table::Common, 1637ff (reading temporary file)
>>
>> sub new_filehandle {
>>     my $fh = shift;
>>     binmode($fh, ":utf8") if $::Variable->{MV_UTF8};
>>     return $fh;
>> }
>>
>> As a side note, avoiding the temporary file altogether in this case 
>> would be
>> a good idea as well :-).
>>
> 
> Rok Ružič produced the attached patch to avoid the temporary file,
> which happened to fix the problem in question.

I just committed the patch to Git:

http://git.icdevgroup.org/?p=interchange.git;a=commitdiff;h=fe4fa39a7cf19031b5e463d55c978b6981a9ae39

Regards
          Racke

-- 
LinuXia Systems => http://www.linuxia.de/
Expert Interchange Consulting and System Administration
ICDEVGROUP => http://www.icdevgroup.org/
Interchange Development Team




More information about the interchange-users mailing list