[ic] Error while interpolating page etc/report: UTF-8 Issue

[Gert van der Spoel]

> -----Original Message-----
> From: interchange-users-bounces at icdevgroup.org [mailto:interchange-
> users-bounces at icdevgroup.org] On Behalf Of IC
> Sent: Tuesday, June 02, 2009 2:24 PM
> To: interchange-users at icdevgroup.org
> Subject: Re: [ic] Error while interpolating page etc/report: UTF-8
> Issue
> 
> 
> 
> > -----Original Message-----
> > From: interchange-users-bounces at icdevgroup.org [mailto:interchange-
> users-
> > bounces at icdevgroup.org] On Behalf Of Stefan Hornburg (Racke)
> > Sent: Monday, June 01, 2009 3:41 PM
> > To: interchange-users at icdevgroup.org
> > Subject: Re: [ic] Error while interpolating page etc/report: UTF-8
> Issue
> >
> > IC wrote:
> > >>> Hi folks,
> > >>>
> > >>> I am running IC 5.6.0 with UTF-8, I get the following error
> message
> > >>> which I
> > >>> think is due to having the GBP £ symbol in the etc/report
> template,
> > the
> > >>> error I get is:-
> > >>>
> > >>> Error during creation of order routing main:
> > >>> Error while interpolating page etc/report:
> > >>> Can't find Unicode property definition "Word" at
> > >>> /usr/local/interchange/lib/Vend/Interpolate.pm line 3513.
> > >>>
> > >>> Does anyone know how I can display the £ symbol in the report
> without
> > >>> getting this error, as it is plain text I cannot use &pound or
> the
> > >>> other
> > >>> variants.
> > >> And did you try without the £ symbol and then the error
> disappeared,
> > just
> > >> to
> > >> be sure it is related?
> > >> Is the £ symbol properly converted to UTF-8?
> > >>
> > >>
> > >> CU,
> > >>
> > >> Gert
> > >
> > > Hi Gert,
> > >
> > > On the main server I removed the £ symbols and didn't see the error
> > again,
> > > however on a backup server running the same version of IC with UTF-
> 8 the
> > > same report template with the £ symbol didn't generate the error. I
> did
> > > however get an error on this server from etc/mail_receipt which
> again
> > seems
> > > to be £ symbol related.
> > >
> > > It would be nice to have the symbols but I could switch to GBP or
> just
> > the
> > > digits.
> >
> > Do you have more than one catalog running on these server(s)?
> > If yes, do they have UTF-8 enabled?
> >
> > Regards
> >            Racke
> 
> 
> After converting all the template files to UTF-8 encoding I came across
> even
> more issues, I started seeing this in the debug when IC was trying to
> send
> emails from the ship_notice template, note all the files that are UTF-8
> encoded seem to start with the 3 odd characters  - is this correct??
> 
> The debug showed:-
> 
> Unable to send mail using /usr/sbin/sendmail
> Message follows:
> 
> 

This is the byteordermark I think:

http://en.wikipedia.org/wiki/Byte-order_mark 

"While UTF-8 does not have byte order issues, a BOM encoded in UTF-8 may
nonetheless be encountered. A UTF-8 BOM is explicitly allowed by the Unicode
standard[1], but is not recommended[2], as it only identifies a file as
UTF-8 and does not state anything about byte order.[3] Many Windows programs
(including Windows Notepad) add BOM's to UTF-8 files. However in Unix-like
systems (which make heavy use of text files for file formats as well as for
inter-process communication) this practice is not recommended, as it will
interfere with correct processing of important codes such as the shebang at
the start of an interpreted script.[4] It may also interfere with source for
programming languages that don't recognise it. For example, gcc reports
stray characters at the beginning of a source file, and in PHP, if output
buffering is disabled, it has the subtle effect of causing the page to start
being sent to the browser, preventing custom headers from being specified by
the PHP script. The UTF-8 representation of the BOM is the byte sequence EF
BB BF, which appears as the ISO-8859-1 characters  in most text editors
and web browsers not prepared to handle UTF-8."

You have to remove this from the files on your unix machine as they do not
offer anything useful.



> Rest of message looked ok.
> 
> I have now had to encode the templates back to ISO-8859.
> 
> To stop the initial problem of "Error while interpolating page
> etc/report:"
> I have had to remove dofile from safeuntrap in interchange.cfg, this
> now
> leaves the following error with one regex but at least things seem to
> be
> working:-
> 
> Safe: 'do "file"' trapped by operation mask at
> /usr/lib/perl5/5.8.8/utf8_heavy.pl line 182.
> Compilation failed in require at /usr/lib/perl5/5.8.8/utf8.pm line 17.
> 
> It seems to be in quite a mess, any suggestions?


I think dofile is supposed to be in SafeUntrap .. I have not heard that
being a solution to anything so far.


 
> Andy.
> 
> 
> 
> 
> 
> _______________________________________________
> interchange-users mailing list
> interchange-users at icdevgroup.org
> http://www.icdevgroup.org/mailman/listinfo/interchange-users