220 24271 <1924515.bGRKb8Vbgl@tjmaciei-mobl4> article Path: news.gmane.org!not-for-mail From: Thiago Macieira Newsgroups: gmane.comp.lang.c++.isocpp.proposals Subject: Re: Text_view: A C++ concepts and range based character encoding and code point enumeration library Date: Mon, 08 Feb 2016 23:02:25 -0800 Lines: 99 Approved: news@gmane.org Message-ID: <1924515.bGRKb8Vbgl@tjmaciei-mobl4> References: <9E5DE55D2546134DACFE4D3C117B36FD1F5955EB@us01wembx1.internal.synopsys.com> <6218281.KrDL742Vuu@tjmaciei-mobl4> Reply-To: std-proposals@isocpp.org NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Trace: ger.gmane.org 1455001366 22404 80.91.229.3 (9 Feb 2016 07:02:46 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 9 Feb 2016 07:02:46 +0000 (UTC) Cc: Zhihao Yuan To: std-proposals@isocpp.org Original-X-From: std-proposals+bncBCB4TK757YBRBC46422QKGQEGMZCC6Q@isocpp.org Tue Feb 09 08:02:38 2016 Return-path: Envelope-to: gclcip-std-proposals@m.gmane.org Original-Received: from mail-lf0-f70.google.com ([209.85.215.70]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1aT2Jp-0005WW-Ia for gclcip-std-proposals@m.gmane.org; Tue, 09 Feb 2016 08:02:37 +0100 Original-Received: by mail-lf0-f70.google.com with SMTP id j99sf59928731lfi.1 for ; Mon, 08 Feb 2016 23:02:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isocpp-org.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:user-agent:in-reply-to :references:mime-version:content-transfer-encoding:content-type :x-original-sender:x-original-authentication-results:reply-to :precedence:mailing-list:list-id:x-spam-checked-in-group:list-post :list-help:list-archive:list-subscribe:list-unsubscribe; bh=AtY7O9QqXQHPnex2Io5m2UM4JfajJfNZWNrRMxj6l7Q=; b=Yk/CGxjS1uam9i65r0nuwa9pTKPX+ZE0NB9z6wjHQc34aH/db1vr9TAPO7Qo8QYlE2 wD/8UZQHFigR7ePVVPN6vqSqjK8M9GGl4ZWgsPtYoqdArYv/dbHPeSgOR/eBrEDaKZi3 G4upWeL1veo711j0WLlzr1bJlHposicwkjRdH5nnXdIqgn3PG15ll84Z97pv7+SSvAGt EcsgxjyXAZS4d186JHwnHBH8Bm9pfpxINi62e/F+ITC+AhVtUL/d1uHlVJa+oTzdjZDi R+CfB+nGjncQjsJ0ODJewOtk3ZmiehHauOI6lq1y8gbfxM29t40z3RlMJK X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:cc:subject:date:message-id:user-agent :in-reply-to:references:mime-version:content-transfer-encoding :content-type:x-original-sender:x-original-authentication-results :reply-to:precedence:mailing-list:list-id:x-spam-checked-in-group :list-post:list-help:list-archive:list-subscribe:list-unsubscribe; bh=AtY7O9QqXQHPnex2Io5m2UM4JfajJfNZWNrRMxj6l7Q=; b=JdmWpkeV1wtnl0r75HA+dR0RxH6Eu+QK++bH8Z/bFHTa6uW7+5eraHMpX+Epem6VwW lDPFWSA3+UJgEbh2RNSfia+v5P8k2dK0mxCBK0OdVdYRNDI9A2wPqJUV+DszEQVRkS2G 75Yd2160+LPeYRaO87YY9sGpRNuBJ18e5ytkbRoWdyJ8r60z6elEw1b9kbKdzr04tVUW 6LOMWFSVPQE/aiQuUJLPW0CdP9RpYQugYb42YjXFSq5+WsbIpYenHqR78371xbw9WyOJ Toe2mKQv7tqiAJ+QfmqRxEzS/jG/W9rHk51VDUCI7B9a6ftUSmaU3Q X-Gm-Message-State: AG10YOSN1Z14PuvAbZqxBoPFUH4lGeq0Cv6/jqb8JkfA2CWC9MXO4XSJ/yTy/6FubmXFRQ== X-Received: by 10.112.132.74 with SMTP id os10mr3867547lbb.22.1455001356752; Mon, 08 Feb 2016 23:02:36 -0800 (PST) X-BeenThere: std-proposals@isocpp.org Original-Received: by 10.28.48.145 with SMTP id w139ls524940wmw.29.gmail; Mon, 08 Feb 2016 23:02:34 -0800 (PST) X-Received: by 10.194.113.102 with SMTP id ix6mr35357517wjb.143.1455001354731; Mon, 08 Feb 2016 23:02:34 -0800 (PST) Original-Received: from gondolin.macieira.info (gondolin.macieira.info. [78.47.120.188]) by mx.google.com with ESMTP id n187si21197220wmb.74.2016.02.08.23.02.34 for ; Mon, 08 Feb 2016 23:02:34 -0800 (PST) Received-SPF: pass (google.com: domain of thiago@macieira.org designates 78.47.120.188 as permitted sender) client-ip=78.47.120.188; Original-Received: from tjmaciei-mobl4.localnet (unknown [IPv6:2601:1c0:5803:81c9:44a5:688d:6e87:ce6a]) by gondolin.macieira.info (Postfix) with ESMTPSA id 78F1611A463; Mon, 8 Feb 2016 23:02:32 -0800 (PST) User-Agent: KMail/5.1.1 (Linux/4.4.0-3-default; KDE/5.18.0; x86_64; ; ) In-Reply-To: X-Original-Sender: thiago@macieira.org X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of thiago@macieira.org designates 78.47.120.188 as permitted sender) smtp.mailfrom=thiago@macieira.org Precedence: list Mailing-list: list std-proposals@isocpp.org; contact std-proposals+owners@isocpp.org List-ID: X-Spam-Checked-In-Group: std-proposals@isocpp.org X-Google-Group-Id: 399137483710 List-Post: , List-Help: , List-Archive: List-Subscribe: , List-Unsubscribe: , Xref: news.gmane.org gmane.comp.lang.c++.isocpp.proposals:24271 Archived-At: On ter=C3=A7a-feira, 9 de fevereiro de 2016 00:25:28 PST Zhihao Yuan wrote: > On Mon, Feb 8, 2016 at 11:32 PM, Thiago Macieira =20 wrote: > > I'd require US-ASCII, Latin 1 (ISO-8859-1) and the "System" encoding as > > mandatory. All others should be optional, probably by way of using ICU'= s > > unicode/ucnv.h functionality. >=20 > Neutral on ASCII, against iso-8859-1 -- implementation-defined > is where it should go. Latin1 is useful because it's a strict 1:1 mapping of UCS-4 to 8 bits, so= =20 transforming from Latin 1 to UTF-16 or UCS-4 is extremely fast. Any "from U= S- ASCII" code could just as well be "from Latin 1". The conversion "To Latin1" is also very easy if you define that the output = is=20 undefined if the source contained non-Latin1 content. It's also the exact s= ame=20 code for "To US-ASCII" if you add the same constraint to it. Converting with error checking to either Latin1 or to US-ASCII is usually a= s=20 difficult, though it varies depending on whether the platform has signed or= =20 unsigned saturation. > > "System" is whatever passes for the current locale in the system. On > > Windows systems, that's the mis-named "ANSI" encoding, like CP 1252. Th= at > > should match the narrow character execution charset. > >=20 > > Strictly speaking, you'll also need a wide-character "System", but I kn= ow > > of no system where wchar_t is not either UTF-16 or UCS-4, so it will be > > just an alias. >=20 > As a FreeBSD user, I have to say... no. >=20 > We use each locale's encoding's documented, official code points in > wchar_t. >=20 > NetBSD uses Japanese standards' code points for Japanese, > UCS-4 for others. Interesting, I didn't know that. But note how the compiler doesn't agree wi= th=20 that and always assumes UCS-4 for the wide character execution charset (and= =20 for that matter, UTF-8 for the source character set): $ uname -sr FreeBSD 10.2-RELEASE-p9 $ export LC_ALL=3Dcs_CS $ cat test.cpp #include #include #include int main() { setlocale(LC_ALL, ""); wprintf(L"=C3=A0\n"); } $ clang test.cpp $ ./a.out | od -tx1 0000000 e0 0a 0000002 $ LC_ALL=3Del_GR ./a.out | od -tx1 0000000 e0 0a 0000002 The compiler converted "=C3=A0" to wchar_t(0x00E0), which the C library in = the=20 Czech locale interprets as "=C5=95". You've got Mojibake. > And of course, IBM doesn't agree with you either, since it doesn't > agree with everything... Yeah, no comment. --=20 Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org Software Architect - Intel Open Source Technology Center --=20 ---=20 You received this message because you are subscribed to the Google Groups "= ISO C++ Standard - Future Proposals" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to std-proposals+unsubscribe@isocpp.org. To post to this group, send email to std-proposals@isocpp.org. Visit this group at https://groups.google.com/a/isocpp.org/group/std-propos= als/. .