From 600073051535791506 X-Google-Language: ENGLISH,ASCII-7-bit X-Google-Thread: f78e5,35a35bec39c95517 X-Google-Attributes: gidf78e5,public X-Google-ArrivalTime: 2001-04-05 09:08:06 PST Path: supernews.google.com!sn-xit-03!supernews.com!cyclone2.usenetserver.com!news-out.usenetserver.com!newsfeed.icl.net!dispose.news.demon.net!news.demon.co.uk!demon!mail2news.demon.co.uk!not-for-mail From: no_spam_va@org.chemie.uni-frankfurt.de (Volker Apelt) Newsgroups: comp.std.c++ Subject: Re: wchar & Unicode Date: Thu, 5 Apr 2001 16:06:33 GMT Organization: UNI - FRANKFURT (Main,germany) Approved: Fergus Henderson , moderator of comp.std.c++ Message-ID: References: <3AB26307.A782192D@spamcop.net> <3AB2C76E.AA9DC137@wizard.net> <3AC80F9A.C929825A@wizard.net> X-Trace: mail2news.demon.co.uk 986486804 mail2news:16337 mail2news mail2news.demon.co.uk X-Complaints-To: abuse@demon.net X-Mail2News-Path: news.demon.net!mulga.cs.mu.oz.au X-Authentication-Warning: mulga.cs.mu.OZ.AU: fjh set sender to devnull@stump.algebra.com using -f X-Robomod: STUMP, ichudov@algebra.com (Igor Chudov) User-Agent: ProNews/2 V1.51.ib102 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-Comments: The University of Frankfurt/Main, Germany does not condone, nor support, spam or any illegal postings, nor copyright infringements. X-Report: Please report illegal or inappropriate use to . X-Abuse-Info: Please send us a complete copy of the offending article, including its full header but exluding any attachments. Lines: 103 Xref: supernews.google.com comp.std.c++:4133 "James Kuyper Jr." wrote: > Volker Apelt wrote: > .... > > So, convenience to me means to have an iterator that knows about > > the current encoding of the string it belongs to and is guaranteed > > to stop at codepoints instead of raw bytes (char) or raw wchar_t. > > I sympathize with your desire, but I think there may be a basic > conceptual flaw with this idea. For iterators, i++ is supposed to > advanced you past the object *i. However, *i is supposed to return > iterator_traits::value_type, while you're suggesting an > iterator that may move you across a different number of bytes, depending > upon the current state. The only way I can see to do that is to have > value_type be either a fixed-size character type such as wchar_t, or > some tricky class type that implicitly converts to a fixed-sized > character type under appropriate circumstances. That's why mbc's are not just working with the standard implementation of string. :-) Step size of multibyte character iterator (mbc_iterator) ++i and size of a raw char of the underlying implementation is not equal. Even worse, size of the next step over a mbc depends on that mbc. ++i is supposed to step over one or more objects of charT, where charT is a type capable of holding the smallest code point for that encoding, probably a byte (octet). So, the implementation of a mbc_iterator is not a simple pointer. It must contain a reference to some encoding information and, if the encoding requires it, some state information. > [ about iterator_traits::value_type and *i return value] That depends on how we will use the mbc_iterator. The only advantage of mbc's over fixed width characters is space consumption. Not converting to wide characters is an advantage, if the amount of mbc's read is huge and you don't need to change the mbc string. eg: searching a small pattern in a large mbc encoded file. So, one could mmap the file, search over it and unmap it without copying or converting. I'd like to use a reduced set of non-changing operations on mbc_iterator: ( i,j,k are mbc_iterator's, w is a wchar_t iterator, c is a char* ) - ++i , i++, i += unsigned, point to the first byte of the next codepoint. maybe it should just throw, in case of partial or error. - testing for validity of the last ++i operation and current position. !i, i.convResult() (Did the last operation step over a complete mbc...?) - compare mbc's to mbc's (*i==*k) (*i!=*k) - (?) compare mbc's to wchar_t and (?) char (*i==*w), (*i==*c) - use non-changing std algorithms like find, find_if on i pairs (rfind could be tricky) It sounds much like an input_iterator. Maybe it requires specialized templates for standard algorithms on mbc_iterators. If --i is implementable, it is a 'bidirectional_input_iterator' (not in std). Is a raw char good enough as iterator_traits::value_type? That was my initial thought. But now I think it is better to make it a readonly proxy, that converts to different char sizes if necessary and throws if it can't. That means the proxy knows about the position and the encoding, too. Or make it a proxy that is accessible for friend funcktions and methods only to restrict access and avoid invisible conversions. But one could argue, that concatenating two mbc strings is efficient,too, if they have the same encoding. But assigning to arbitrary positions isn't required to. Concatenation may require some additional bytes, which are not present in the original strings, to adjust for different shift states. That requires at least: - *i=*k, *i=*w, *i=*c, should be fast if i == end() Then, iterator_traits::value_type probably has to be a complicated proxy type, if assignment is required. ( This is not what I need. ) > I suppose you could define a string whose iterators present a value_type > of wchar_t, but make use of codecvt on an purely internal multibyte > character string. However, that would leave the multibyte character > string almost completely hidden; I suspect that's not what you want. That sounds interesting, too, but could be a completely different iter type. My initial idea was an implementation, which allows to access the raw representation and does as little work as possible (no conversion). The proxy above could do lazy evaluation and cache a conversion result for the last character. But 'doing no conversion' may be an illusion. I'm not sure what the correct type for value_t would be. I favour the proxy implementation. --- [ comp.std.c++ is moderated. To submit articles, try just posting with ] [ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.edu ] [ --- Please see the FAQ before posting. --- ] [ FAQ: http://www.research.att.com/~austern/csc/faq.html ]