From -8404472872805813157 X-Google-Language: ENGLISH,ASCII-7-bit X-Google-Thread: f78e5,b1fa62568eb1d2a7 X-Google-Attributes: gidf78e5,public X-Google-ArrivalTime: 2002-02-13 11:43:01 PST Path: archiver1.google.com!news1.google.com!newsfeed.stanford.edu!newsfeeds.belnet.be!news.belnet.be!colt.net!dispose.news.demon.net!news.demon.co.uk!demon!mail2news.demon.co.uk!not-for-mail From: "Ken Shaw" Newsgroups: comp.std.c++ Subject: Re: Unicode vs. Char Date: Wed, 13 Feb 2002 19:41:53 GMT Organization: Concentric Internet Services Approved: Fergus Henderson , moderator of comp.std.c++ Message-ID: References: <0hU68.89357$h31.5940930@e420r-atl1.usenetserver.com> X-Trace: mail2news.demon.co.uk 1013629327 mail2news:25561 mail2news mail2news.demon.co.uk X-Complaints-To: abuse@demon.net X-Mail2News-Path: news.demon.net!mulga.cs.mu.oz.au X-Authentication-Warning: mulga.cs.mu.OZ.AU: fjh set sender to devnull@stump.algebra.com using -f X-Robomod: STUMP, ichudov@algebra.com (Igor Chudov) Errors-To: X-Priority: 3 X-MSMail-Priority: Normal X-Newsreader: Microsoft Outlook Express 6.00.2600.0000 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2600.0000 Lines: 106 Xref: archiver1.google.com comp.std.c++:9595 "Hans Aberg" wrote in message news:remove.haberg-1302021239280001@du137-226.ppp.su-anst.tninet.se... > In article , "Robert Buck" > wrote: > >> Or one may view Unicode as a part of the distributed programming package > >> (see first article in thread "C++0x" last year): It turns out that the > >> current C++ Unicode support isn't portable, so that programmers that need > >> that feature end up writing out the names of the characters and their > >> Unicode values. This is extremely tedious, especially now when Unicode may > >> have hundreds of thousands of characters. > > > >Huh? Why would you think engineers would have to jump through hoops. > >Any engineer worth their weight will use a commercial library, such as ICU, > >that manages LE / BE issues for you, performs transcodings, > >transliterations, > >and word/sentence boundary analysys. No worries. > > There was some guy in this group writing a multicompiler WWW browser > and/or server, who said he ended up explicitly writing out the names of > Unicode characters as identifiers, and their encoding numbers. > > C++ does have some "Unicode" strings, but it turns out that they do not > guarantee to produce any Unicode characters. > > Of course, if somebody writes a multiplatform library, working through all > the compilers in existence, that is one way around it. Another would be to > put it into the standard. > > >> Otherwise, everybody expects Unicode is created by a consortium trying to > >> make all other characters encodings unnecessary. There is room for > >> introducing user characters on top of the Unicode range (between 2^21 and > >> 2^24-1 I think). So if you have Unicode, and what for what reason would > >> you use another character encoding when communicating with people? > >> (Disregarding the fact that it might be useful with special compacted > >> encodings for special purposes -- but that would not be used in order to > >> guarantee open communications.) > > > >If you stuck to UTF-32 or UTF-16 _only_, how might you write a real world > >business application that aggregates data from disparate business systems > >over sockets? That is where encodings such as UTF-8 come into place, > >and where commercial Unicode libraries that have transcoders come into > >place. > > I think the quote got out of context: UTF-32 will probably be the only one > to adhere to _internally_ in newly written programs that handle many > potential different characters. Then hook any codecvt for use with other > external encodings. > > >Any time you deal with wire-protocols you are stuck with dealing with > >bytes, not 16-bit or even 32-bit units of information. > > If there is no such protocol for UTF-32, perhaps there should be. > > > Choose what makes > >sense for the application. Suppose you are getting data from VSAM/ISAM > >files, > >and that data is returned in EBCDIC format, what do you do? Tell the > >customer to upgrade to Unicode? > > If the new program uses UTF-32 internally, hook onto a suitable > std::codecvt, I think. > > >I worked for an object-database company on a pure XML database. As > >with many databases on the market, character data was actually stored in > >UTF-8. All operations also were performed on UTF-8. > > When storing information like on a file or in a data base, or transporting > over the Internet, one may use some kind of compressions scheme. UTF-8 > might be viewed as such a compressions scheme relative UTF-32. > > But I do not see why one should use UTF-8 as the one and only compression > scheme: Perhaps other schemes will work much more efficiently. > One very good reason for using UTF-8 is that legacy software will work if fed UTF-8 while most other encodings (any that allow a byte in the stream to have a value of 0) will cause those systems to fail. I would be all in favor of requiring wchar_t to be 32 bits and requiring that the standard library include the appropriate UTF-8 to UTF-32 codecvt specializations. This would satisfy quite a lot of i18n needs without adding all the complexities of the ICU to the standard library. Ken Shaw -- The tree of liberty must be refreshed from time to time with the blood of patriots and tyrants. It is its natural manure. Thomas Jefferson, 1787 --- [ comp.std.c++ is moderated. To submit articles, try just posting with ] [ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.edu ] [ --- Please see the FAQ before posting. --- ] [ FAQ: http://www.research.att.com/~austern/csc/faq.html ]