220 10556 <0845BE08-100D-41C1-9AA1-7CCFAFB96C67@gmail.com> article Path: news.gmane.org!not-for-mail From: Dietmar Kuehl Newsgroups: gmane.comp.lang.c++.isocpp.proposals Subject: Re: Re: Unicode support in the Standard Library Date: Thu, 8 May 2014 23:28:53 +0100 Lines: 143 Approved: news@gmane.org Message-ID: <0845BE08-100D-41C1-9AA1-7CCFAFB96C67@gmail.com> References: <4ef82544-cd98-4488-8230-88ddaea78562@isocpp.org> <002D029C-6783-4A62-8CC4-B32B7BE8B23D@gmail.com> <01f2cc36-e743-4782-891e-c074dc072c7f@isocpp.org> <0c701b5e-60b2-4940-9fcd-d86b1d75a9db@isocpp.org> Reply-To: std-proposals@isocpp.org NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 (1.0) Content-Type: multipart/alternative; boundary=Apple-Mail-2FECCA83-3F0D-4C03-A592-445411E943A2 Content-Transfer-Encoding: 7bit X-Trace: ger.gmane.org 1399588149 32345 80.91.229.3 (8 May 2014 22:29:09 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Thu, 8 May 2014 22:29:09 +0000 (UTC) Cc: "mw_triad@users.sourceforge.net" To: "std-proposals@isocpp.org" Original-X-From: std-proposals+bncBDZYDW6QHYIKXCVQTMCRUBCTQDTCC@isocpp.org Fri May 09 00:29:01 2014 Return-path: Envelope-to: gclcip-std-proposals@m.gmane.org Original-Received: from mail-ee0-f72.google.com ([74.125.83.72]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1WiWoG-0008S2-UD for gclcip-std-proposals@m.gmane.org; Fri, 09 May 2014 00:29:01 +0200 Original-Received: by mail-ee0-f72.google.com with SMTP id d49sf2064500eek.7 for ; Thu, 08 May 2014 15:29:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:references:mime-version:in-reply-to:message-id :cc:from:subject:date:to:x-original-sender :x-original-authentication-results:reply-to:precedence:mailing-list :list-id:list-post:list-help:list-archive:list-subscribe :list-unsubscribe:content-type:content-transfer-encoding; bh=3LowV5K2MaGYi8XJ0fXdZAjIPvIT08m/sqmIKcdNsdA=; b=J0qaP1rp9fKBrzNBgCGkJOIaUI92llWqdVh6cJIO6DQxUQ56krFEW5KBNhrc3J/Iv9 sh9dbp+o65m5UwPqs8WzPyQQeOwlFBqPbxQaDJkY1cLAo7cmezF/tB6181H2M9nSJLqW 8jNo0/XlYGSZchDQZT57Mh//O86PQLqXasWjEIpj6W95HZPZf9FgXWz4KyBGTbegS1nm xiwRKfo1e5Cg3UPw2+ozK92L/AY3r+fU9Ppojal8eVpI32tFDugNEcmwr94dKVkacHSg b0rj5MfoWgoONkox4gw3wB885cZt37KWBtahVA1W16BdH87kxbl72RH4oUAHlKMZy87J 4BrQ== X-Gm-Message-State: ALoCoQmDCc/OH2N5DtBy/da4EO8NJMMRSKbADg14cKGpQ5HdZs9gtgpMX7mlaVmdewl8ebDp6Px1 X-Received: by 10.112.9.168 with SMTP id a8mr760886lbb.10.1399588140419; Thu, 08 May 2014 15:29:00 -0700 (PDT) X-BeenThere: std-proposals@isocpp.org Original-Received: by 10.180.92.98 with SMTP id cl2ls264155wib.53.canary; Thu, 08 May 2014 15:28:58 -0700 (PDT) X-Received: by 10.180.98.231 with SMTP id el7mr512281wib.1.1399588138767; Thu, 08 May 2014 15:28:58 -0700 (PDT) Original-Received: from mail-wg0-x22e.google.com (mail-wg0-x22e.google.com [2a00:1450:400c:c00::22e]) by mx.google.com with ESMTPS id r9si252257wia.103.2014.05.08.15.28.58 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 08 May 2014 15:28:58 -0700 (PDT) Received-SPF: pass (google.com: domain of dietmar.kuehl@gmail.com designates 2a00:1450:400c:c00::22e as permitted sender) client-ip=2a00:1450:400c:c00::22e; Original-Received: by mail-wg0-f46.google.com with SMTP id n12so3106962wgh.17 for ; Thu, 08 May 2014 15:28:58 -0700 (PDT) X-Received: by 10.180.36.66 with SMTP id o2mr316771wij.40.1399588138283; Thu, 08 May 2014 15:28:58 -0700 (PDT) Original-Received: from [192.168.0.7] (cpc18-dals16-2-0-cust21.20-2.cable.virginm.net. [81.100.53.22]) by mx.google.com with ESMTPSA id q9sm2945557wjo.3.2014.05.08.15.28.56 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 08 May 2014 15:28:57 -0700 (PDT) In-Reply-To: X-Mailer: iPhone Mail (11D201) X-Original-Sender: dietmar.kuehl@gmail.com X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of dietmar.kuehl@gmail.com designates 2a00:1450:400c:c00::22e as permitted sender) smtp.mail=dietmar.kuehl@gmail.com; dkim=pass header.i=@gmail.com; dmarc=pass (p=NONE dis=NONE) header.from=gmail.com Precedence: list Mailing-list: list std-proposals@isocpp.org; contact std-proposals+owners@isocpp.org List-ID: X-Google-Group-Id: 399137483710 List-Post: , List-Help: , List-Archive: List-Subscribe: , List-Unsubscribe: , Xref: news.gmane.org gmane.comp.lang.c++.isocpp.proposals:10556 Archived-At: --Apple-Mail-2FECCA83-3F0D-4C03-A592-445411E943A2 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Sadly, Unicode isn't as simple: there are multiple representation of charac= ters which are equivalent. For example the u-umlaut (=C3=BC) in my name can= be represented as u-umlaut or as a u and a combining character dieresis. T= o match a u you'll need to understand whether the preceding (I think) chara= cter is a combining character or not. To do so you need to understand the e= ncoding. Unicode had promised to make that simple by having three fundamental design= rules: - every character is represented by one code point; that went over board an= d there are now combining characters - every string has a unique representation as a sequence of code points; th= at's not true as you can choose certain orders of characters - each code point uses 16 bits; well, last time I looked they were at 20 bi= ts If they had stuck with their original goals things would be nearly as simpl= e as you'd think. As is, they are not. > On 8 May 2014, at 19:49, Diggory Blake wrote: >=20 >=20 >=20 >> On Thursday, 8 May 2014 19:44:30 UTC+1, Matthew Woehlke wrote: >>=20 >> Please pardon me showing my unicode ignorance here, but are even search= =20 >> operations a problem? That is, if I search for e.g. '=C3=A1', can it eve= r=20 >> match a second (or third or...) byte of some other multi-byte character?= =20 >>=20 >> I want to say 'no', in which case even search (or substring compare)=20 >> operations would not need to care about encoding... basically, only n'th= =20 >> character operations and iterating over characters. (With the caveat=20 >> that the result you get back is a byte index and not a character index.) >=20 > It depends on the encoding - utf8 for example ensures that the representa= tion of one complete code-point can never occur as part of another > --=20 >=20 > ---=20 > You received this message because you are subscribed to the Google Groups= "ISO C++ Standard - Future Proposals" group. > To unsubscribe from this group and stop receiving emails from it, send an= email to std-proposals+unsubscribe@isocpp.org. > To post to this group, send email to std-proposals@isocpp.org. > Visit this group at http://groups.google.com/a/isocpp.org/group/std-propo= sals/. --=20 ---=20 You received this message because you are subscribed to the Google Groups "= ISO C++ Standard - Future Proposals" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to std-proposals+unsubscribe@isocpp.org. To post to this group, send email to std-proposals@isocpp.org. Visit this group at http://groups.google.com/a/isocpp.org/group/std-proposa= ls/. --Apple-Mail-2FECCA83-3F0D-4C03-A592-445411E943A2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Sadly, Unicode isn't as simple: th= ere are multiple representation of characters which are equivalent. For exa= mple the u-umlaut (=C3=BC) in my name can be represented as u-umlaut or as = a u and a combining character dieresis. To match a u you'll need to underst= and whether the preceding (I think) character is a combining character or n= ot. To do so you need to understand the encoding.

= Unicode had promised to make that simple by having three fundamental design= rules:

- every character is represented by one co= de point; that went over board and there are now combining characters
=
- every string has a unique representation as a sequence of code point= s; that's not true as you can choose certain orders of characters
- each code point uses 16 bits; well, last time I looked they were at 20 b= its

If they had stuck with their original goals things would be near= ly as simple as you'd think. As is, they are not.

On 8 May 20= 14, at 19:49, Diggory Blake <d= iggsey@googlemail.com> wrote:



On Thursday, 8 May 2014 19:44:30 UTC+1, Matt= hew Woehlke wrote:

Please pardon me showing my unicode ignorance here, but are even search= =20
operations a problem? That is, if I search for e.g. '=C3=A1', can it ev= er=20
match a second (or third or...) byte of some other multi-byte character= ?

I want to say 'no', in which case even search (or substring compare)=20
operations would not need to care about encoding... basically, only n't= h=20
character operations and iterating over characters. (With the caveat=20
that the result you get back is a byte index and not a character index.= )

It depends on the encoding - utf8 for example ens= ures that the representation of one complete code-point can never occur as = part of another

--

---
You received this message because you are subscribed to the Google Groups "= ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to std-proposa= ls+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
Visit this group at http://groups.google.com/a/isocpp.org/group/std-proposals/<= /a>.

--

---
You received this message because you are subscribed to the Google Groups &= quot;ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to std-proposa= ls+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
Visit this group at http://groups.google.com/a/isocpp.org/group/std-proposals/<= /a>.
--Apple-Mail-2FECCA83-3F0D-4C03-A592-445411E943A2-- .