220 37533 article Path: news.gmane.org!.POSTED!not-for-mail From: =?UTF-8?B?0JTQuNC80LjRgtGA0LjRmCDQnNC40ZjQvtGB0LrQuA==?= Newsgroups: gmane.comp.lang.c++.isocpp.proposals Subject: Re: Re: Unicode support by extending std::locale. Can we make it by 2020? Date: Wed, 28 Mar 2018 17:11:41 +0200 Lines: 255 Approved: news@gmane.org Message-ID: References: <45303792-68f2-4545-8ce4-4a3e1ec35b1b@isocpp.org> <8174836d-21fd-4030-aee9-bcb43d83d0fb@isocpp.org> Reply-To: std-proposals@isocpp.org NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="001a11451f08ab7ae905687a6c5f" X-Trace: blaine.gmane.org 1522249781 12741 195.159.176.226 (28 Mar 2018 15:09:41 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Wed, 28 Mar 2018 15:09:41 +0000 (UTC) To: std-proposals@isocpp.org Original-X-From: std-proposals+bncBCRIFJHGXANBBLXB53KQKGQE4UTXQCQ@isocpp.org Wed Mar 28 17:09:37 2018 Return-path: Envelope-to: gclcip-std-proposals@m.gmane.org Original-Received: from mail-qk0-f197.google.com ([209.85.220.197]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1f1Chk-0003Ca-Bm for gclcip-std-proposals@m.gmane.org; Wed, 28 Mar 2018 17:09:36 +0200 Original-Received: by mail-qk0-f197.google.com with SMTP id 86sf1789109qkr.22 for ; Wed, 28 Mar 2018 08:11:44 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1522249903; cv=pass; d=google.com; s=arc-20160816; b=XmIC5lZf7wLWT8dq6y8zzyF2w/GjO6KUQc6ltun9xjEce31BrJ0hinmo/mMqkL3agm VJNcjYN3zTHs0VUnokTQHS1dHpOQPzgga6FJcZ5aUsWheXOli3wZtfuWvKbxPQ13I8yD pIyUqaj79iq6WQx69k2iI57YqCvD413yX5JZ9gCwAPM49hP9SS1I1DZj4C4cJkBPGuu+ 3iieDu7KlbXGqnmeh90LfWabzibiUi0iL0B4oNxFYBUJQFBHRs6um6AiKTFrl0LXkdqo CunaCSojEpALI+jCMKqD0JIQJxVTjmBne7g0d4xBqBs/mVUQnRo2Leyal73oLKJK+NEY gjrw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-unsubscribe:list-subscribe:list-archive:list-help:list-post :list-id:mailing-list:precedence:reply-to:to:subject:message-id:date :from:references:in-reply-to:mime-version:arc-authentication-results :arc-message-signature:dkim-signature:arc-authentication-results; bh=ul59bsNhAqF6xP0eIEFnjWVryyRr2y+DXbzhGPzPAJE=; b=DLW7+MwC5L3RFIx9TSw3MtuXvQJwdakCiZMCZOEyzCmeeC8cDCQhb938OZG7oaYajY Vwi4KCLqNjuK+LtmZeBIkxpZev400wfslXVWk8uZ2eKJxTZKepEc2qjL8++0Vt0yi1xR /hL2M3ay9EvqAkZIun4Ky7KIOcCtlnngra/O4efsKJtKB7++odS8IKhR/PnEva5/lgA2 ugvuioDEj/n+Ihj9PJZRmwL1IakRMN2V4cw+zDKO0qj4+KFdo6DaGwuekXhhlDds3SNW OCDStu5KevvJ8JqrKySmEr/kvC4HWiB7z6V+m2z6Sj1jtoSMaJliwbukTpKXdYKxqHZz D18w== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=NiJDBPII; spf=pass (google.com: domain of dim.mj.p@gmail.com designates 209.85.220.41 as permitted sender) smtp.mailfrom=dim.mj.p@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isocpp-org.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :x-original-sender:x-original-authentication-results:reply-to :precedence:mailing-list:list-id:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=ul59bsNhAqF6xP0eIEFnjWVryyRr2y+DXbzhGPzPAJE=; b=UbdScAxrjfhFOBV/DtkjyJmsCs+b2Rc1FimO2MfcO1bnHeKoJyCP5HCyzSLTWZL/S0 MOGUeTgj/QQ2hXZBkmEXpGbPLd9GvfEAiMH9djnXB3u9SsgwtGTalkIxADrnytuY7psl BtN6LDb+y+bNdRtivY8T00QHVVdGHVrpKDp5u3PS2YIdZp7luD/vKYRMlSY12bjWOf+u mG3pMj4xTHDvBaM73DvHMjFA+yg1nU0EIC849rXlSwRxxCb2zxpzq2ciuWzi53omLhJA c2ZG2SszzNI/C2bNbqzwGGHnqDkLWOByAVk5wKoeLFT7aAvD899VrZkfwv37GNPhcwtZ pCnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:x-original-sender :x-original-authentication-results:reply-to:precedence:mailing-list :list-id:x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=ul59bsNhAqF6xP0eIEFnjWVryyRr2y+DXbzhGPzPAJE=; b=HYxzCO5daH9JKCZB5EtYigZVoSZscXYA2j4re/q9/Eb3JAPHUwZz7W01nwZEWL7+bx Q7x5uheZ0Z88xgvPidGKaoDoWkuSi3sO7saSWkVkEXpmdf71tBgh1LTc+vznSFiE2787 ehK0tQy7N2z+TzC1zGGYl1yyZ7Xz1B9lyCG3vmzNKkrUeQ9WN5Fj8ZQfOEh9FVbnS7EN 8O0Ovh6m3yVlxRGHZglWzRo2Ul4tIJgkN1g2iQq3Afcyl2mIIKRqB2kvbZe9qK97GkEO ylnr5dHHT4LibAt8qi9MQ3Izny063QRbxwlc9S13DpHvdEscN5kUigKZ8Xh4NgBAlj09 7qMA== X-Gm-Message-State: AElRT7GLOWc4B1swBNv0qgOO7v7ANkQR5iRj1a4lROqV358viWz69yoW hWaOmVgUJS80CMb5UzvSB3GKVw== X-Google-Smtp-Source: AG47ELso9lOpKlsGu+J+gnHv3JSBg5S/9353bLgSZmg/b70b7KzskDKy2UTfRbKw5cGdBA9SNopJCA== X-Received: by 10.200.3.211 with SMTP id z19mr25200449qtg.14.1522249903593; Wed, 28 Mar 2018 08:11:43 -0700 (PDT) X-BeenThere: std-proposals@isocpp.org Original-Received: by 10.55.120.193 with SMTP id t184ls1627267qkc.13.gmail; Wed, 28 Mar 2018 08:11:42 -0700 (PDT) X-Received: by 10.55.1.194 with SMTP id u63mr5661630qkg.47.1522249902470; Wed, 28 Mar 2018 08:11:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1522249902; cv=none; d=google.com; s=arc-20160816; b=H7HmXYYNkI3t+79YOYFDKCD51wf0G8omeaVE1uie4B9nf67xylt6D2bOtNS1tcLOX8 Yia5/+/aEMOmbvy3ON/5REMm6oF1Fu/gggbHe/eESGPqfB7LxR60MS4WlSxxeZwUbnqd Z9bRtu5qw3nqtAVeAVNcObQ2R1AckK2d4HpBe82e9JqZOwKiEcSw0bc2Er06sY1qdd97 AzGI54zU+fxS9wPRdcG0kFSYx3PlmZxBHx0IqjKxiImuKBLYDvVD7/ou9+Mtp6geMvs0 mnvhRx0IQrP6aZOEcavXLnAzdcFrWrYpADaBMCrA6nmFTo3mPwZi+v8FujRIn3hPRlzu ZMSQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=to:subject:message-id:date:from:references:in-reply-to:mime-version :dkim-signature:arc-authentication-results; bh=eIqQdN4fzRqPSTOXx6mxzgFBZe9DVQn7Z7zsWzLBwtI=; b=Z/5G0v6fDFHcJC51B/OFxBSFk0qhU/CPCEmZclRV0vYgO1Xv8ClLUQnQTSKV8IbX0m WqvUH/h2a96IF730EbwvichgXb2pypqsjo/jZQ5Qq4zVOWW/+RXDg7Z6OoOrDCzUIVxA /qlypiT8Z94xO5QIu2ptxzSwxKycnl3MqeWEEIFGWvVwCg+S3OI9G6gClwpFSASpVZ3A NSOIo4uMsiC4JEXqIPYUogb/xnZ1sRkSbCtphi6yFSYjroP4Cet/VzgbZtbsW2ivpQAF 0J+iuXWoNm7XgkmeXPPwzwgDfnkkhYxAM+yh1gLe7hTu7T4VI5YELuQ29IZNCMfjEXYv W+6A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=NiJDBPII; spf=pass (google.com: domain of dim.mj.p@gmail.com designates 209.85.220.41 as permitted sender) smtp.mailfrom=dim.mj.p@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Original-Received: from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41]) by mx.google.com with SMTPS id x20sor3664998qtb.87.2018.03.28.08.11.42 for (Google Transport Security); Wed, 28 Mar 2018 08:11:42 -0700 (PDT) Received-SPF: pass (google.com: domain of dim.mj.p@gmail.com designates 209.85.220.41 as permitted sender) client-ip=209.85.220.41; X-Received: by 10.200.9.121 with SMTP id z54mr5696081qth.87.1522249902028; Wed, 28 Mar 2018 08:11:42 -0700 (PDT) Original-Received: by 10.12.134.102 with HTTP; Wed, 28 Mar 2018 08:11:41 -0700 (PDT) In-Reply-To: <8174836d-21fd-4030-aee9-bcb43d83d0fb@isocpp.org> X-Original-Sender: dim.mj.p@gmail.com X-Original-Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=NiJDBPII; spf=pass (google.com: domain of dim.mj.p@gmail.com designates 209.85.220.41 as permitted sender) smtp.mailfrom=dim.mj.p@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Precedence: list Mailing-list: list std-proposals@isocpp.org; contact std-proposals+owners@isocpp.org List-ID: X-Spam-Checked-In-Group: std-proposals@isocpp.org X-Google-Group-Id: 399137483710 List-Post: , List-Help: , List-Archive: List-Subscribe: , List-Unsubscribe: , Xref: news.gmane.org gmane.comp.lang.c++.isocpp.proposals:37533 Archived-At: --001a11451f08ab7ae905687a6c5f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable AFAIK Unicode defines both simple 1-to-1 case transformations at character level and more complex language sensitive and context sensitive case transformations at string level. You can not just care about the second and throw away the first. The ctype would handle the first. On Wed, Mar 28, 2018 at 4:55 PM, wrote: > I find that there are several important issues in this proposal that nee= d > to be addressed. > > On Wednesday, March 28, 2018 at 2:45:40 PM UTC+2, Dimitrij Mijoski wrote: >> >> Goals: >> >> - [...] >> - Reuse std::locale and facet interfaces >> >> This goal is contradictory with the goal of Unicode support. Several of > these interfaces are simply not suitable for Unicode support (and IMO > should be deprecated). Some of the textbook counterexamples are right the= re > in the code sample provided. `charT toupper(charT, locale)` just cannot > possibly correctly uppercase `=C3=9F` into `SS` (which AFAIK is still the > correct way to uppercase this according to the CLDR locales). Even if the > locale is changed so that it uppercases to `=E1=BA=9E` (following last ye= ar's > decision of the Rat f=C3=BCr deutsche Rechtschreibung, which makes it an = option, > but not a requirement), it's still impossible to uppercase some hundred > other characters, e.g. U+01F0 LATIN SMALL LETTER J WITH CARON. The > fundamental assumption existing in the locale interface is that case > mapping is a 1:1 mapping, but that isn't true. > >> >> - See boost::locale which extends std::locale. >> >> Note that the Boost.Locale documentation even acknowledges the problem I > described above: "You may notice that there are existing functions to_upp= er > and to_lower in the Boost.StringAlgo library. The difference is that thes= e > function operate over an entire string instead of performing incorrect > character-by-character conversions." > > >> - Unicode - a standard that combines ~ 1 million characters into >> single set, then maps each character into unique integer and defines = couple >> of encodings. Namely: UTF-32, UTF-16, and UTF-8. Then defines byte >> serialization of UTF-16 and UTF-32 as UTF-16-BE, UTF-16-LE, UTF-32-BE= and >> UTF-32-LE. >> >> Nitpick: Note that the Unicode Standard defines a lot more than > characters and encodings. > > >> 3. Future proposal >> >> >> ctype >> >> We should completely avoid this gotcha and make ctype work out >> of the box for the whole Unicode range. The locale name should modify on= ly >> the widen() and narrow() functions. >> >> > As mentioned above, this is not enough because the interface itself is > unsuitable for this purpose. > > >> Defining this facet will automatically enable decent Unicode regexes. >> > > This is really debatable. The only thing that `char32_t` gives is the > ability to match on code points instead of matching on code units (which = is > a disaster with `char16_t` and `char`). However, this isn't enough for ev= en > regular expression Level 1 Conformance, because the facilities in > are currently unsuited for this purpose. > > >> ctype >> >> This should behave exactly same as the above, except that it will accept >> only the first 65536 characters of Unicode, i.e. characters from the bas= ic >> multilingual plane (BMP). >> > > This is just designing for deprecation. This ctype would prove entirely > useless for UTF-16, for example. It's essentially UCS-2-only. Pretending > UCS-2 is relevant is the same kind of mistake that made. This > isn't "Unicode support"; it's "Unicode subset support". It's wishful > thinking that people don't use, e.g. the Supplementary Ideographic Plane, > mathematical symbols, or, heck, emoji. Let's not do that again. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "ISO C++ Standard - Future Proposals" group. > To unsubscribe from this topic, visit https://groups.google.com/a/ > isocpp.org/d/topic/std-proposals/Besva70LN3c/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > std-proposals+unsubscribe@isocpp.org. > To post to this group, send email to std-proposals@isocpp.org. > To view this discussion on the web visit https://groups.google.com/a/ > isocpp.org/d/msgid/std-proposals/8174836d-21fd-4030- > aee9-bcb43d83d0fb%40isocpp.org > > . > --=20 You received this message because you are subscribed to the Google Groups "= ISO C++ Standard - Future Proposals" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to std-proposals+unsubscribe@isocpp.org. To post to this group, send email to std-proposals@isocpp.org. To view this discussion on the web visit https://groups.google.com/a/isocpp= ..org/d/msgid/std-proposals/CAORbL%2BMw%3DVCaPuOvo-4NcekDSPgsvHXA6v2QD0wmFUf= C8OgOsw%40mail.gmail.com. --001a11451f08ab7ae905687a6c5f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
AFAIK Unicode defines both simple 1-to-1 case transformati= ons at character level and more complex language sensitive and context sens= itive case transformations at string level. You can not just care about the= second and throw away the first. The ctype<char32_t> would handle th= e first.

On Wed, Mar 28, 2018 at 4:55 PM, <martinho.fernan= des@native-instruments.de> wrote:
=C2=A0I find that there are several important issue= s in this proposal that need to be addressed.

On Wednesday, March 28= , 2018 at 2:45:40 PM UTC+2, Dimitrij Mijoski wrote:
Goals:
  • [...]
  • Reuse std::locale and facet inte= rfaces
This goal is contradictory w= ith the goal of Unicode support. Several of these interfaces are simply not= suitable for Unicode support (and IMO should be deprecated). Some of the t= extbook counterexamples are right there in the code sample provided. `charT= toupper(charT, locale)` just cannot possibly correctly uppercase `=C3=9F` into `SS` (which AFAIK is= still the correct way to uppercase this according to the CLDR locales). Ev= en if the locale is changed so that it uppe= rcases to `=E1=BA=9E` (following last year's decision of the Rat= f=C3=BCr deutsche Rechtschreibung, which makes it an option, but not a req= uirement), it's still impossible to uppercase some hundred other charac= ters, e.g. U+01F0 LATIN SMALL LETTER J WITH CARON. The fundamental assumpti= on existing in the locale interface is that case mapping is a 1:1 mapping, = but that isn't true.
  • See boost::locale which ext= ends std::locale.
Note that the Boo= st.Locale documentation even acknowledges the problem I described above: &q= uot;You may notice that there are existing functions to_upper and to_lower = in the Boost.StringAlgo library. The difference is that these function oper= ate over an entire string instead of performing incorrect character-by-char= acter conversions."

  • Unicode - a standard that comb= ines ~ 1 million characters into=20 single set, then maps each character into unique integer and defines=20 couple of encodings. Namely: UTF-32, UTF-16, and UTF-8. Then defines=20 byte serialization of UTF-16 and UTF-32 as UTF-16-BE, UTF-16-LE,=20 UTF-32-BE and UTF-32-LE.
Nitpick: N= ote that the Unicode Standard defines a lot more than characters and encodi= ngs.
=C2=A0

3. Future proposal


ctype<char32_t>


We should comp= letely avoid this gotcha and make ctype<char32_t> work o= ut of the box for the whole Unicode range. The locale name should modify on= ly the widen() and narrow() functions.


As mentioned above, this is not e= nough because the interface itself is unsuitable for this purpose.
=C2= =A0

Defining this facet will automatically enable decent Unicode re= gexes.


This is really debatable. The = only thing that `char32_t` gives is the ability to match on code points ins= tead of matching on code units (which is a disaster with `char16_t` and `ch= ar`). However, this isn't enough for even regular expression Level 1 Co= nformance, because the facilities in <regex> are currently unsuited f= or this purpose.
=C2=A0

ctype<char16_t>

This should behave exactly same as the above, except that it will=20 accept only the first 65536 characters of Unicode, i.e. characters from=20 the basic multilingual plane (BMP).


T= his is just designing for deprecation. This ctype would prove entirely usel= ess for UTF-16, for example. It's essentially UCS-2-only. Pretending UC= S-2 is relevant is the same kind of mistake that <codecvt> made. This= isn't "Unicode support"; it's "Unicode subset suppo= rt". It's wishful thinking that people don't use, e.g. the Sup= plementary Ideographic Plane, mathematical symbols, or, heck, emoji. Let= 9;s not do that again.

--
You received this message because you are subscribed to a topic in the Goog= le Groups "ISO C++ Standard - Future Proposals" group.
To unsubscribe from this topic, visit https://groups.google.com/a/isocpp.org/d/topic/std-proposals/B= esva70LN3c/unsubscribe.
To unsubscribe from this group and all its topics, send an email to std-prop= osals+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
 To view this discussion on the web visit = https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/8174= 836d-21fd-4030-aee9-bcb43d83d0fb%40isocpp.org.

--
You received this message because you are subscribed to the Google Groups &= quot;ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to std-proposa= ls+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAORbL%2BMw%3= DVCaPuOvo-4NcekDSPgsvHXA6v2QD0wmFUfC8OgOsw%40mail.gmail.com.
--001a11451f08ab7ae905687a6c5f-- .