220 26403 <4d975edd-1c9c-468d-b852-28e2a2108931@isocpp.org> article Path: news.gmane.org!not-for-mail From: Nicol Bolas Newsgroups: gmane.comp.lang.c++.isocpp.proposals Subject: Re: iswalpha and locales Date: Sun, 26 Jun 2016 06:08:44 -0700 (PDT) Lines: 137 Approved: news@gmane.org Message-ID: <4d975edd-1c9c-468d-b852-28e2a2108931@isocpp.org> References: <5057e854-b2ea-4c39-81c7-367bc3e54080@isocpp.org> <84d2b9e6-a29b-4c79-8646-2f00dbc5087b@isocpp.org> <1550791.ovymrLsGO4@tjmaciei-mobl1> <84d2f2a0-0fc4-4526-9826-8062bb443869@isocpp.org> <8ef0b8d3-3c57-4e54-8478-b81c2bd4207a@isocpp.org> Reply-To: std-proposals@isocpp.org NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_163_695688364.1466946524785" X-Trace: ger.gmane.org 1466946529 29042 80.91.229.3 (26 Jun 2016 13:08:49 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sun, 26 Jun 2016 13:08:49 +0000 (UTC) Cc: asorenji@gmail.com To: ISO C++ Standard - Future Proposals Original-X-From: std-proposals+bncBCEKFTV6ZUMBBXVHX65QKGQE6EWR6GA@isocpp.org Sun Jun 26 15:08:49 2016 Return-path: Envelope-to: gclcip-std-proposals@m.gmane.org Original-Received: from mail-it0-f70.google.com ([209.85.214.70]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1bH9ns-00053a-Ns for gclcip-std-proposals@m.gmane.org; Sun, 26 Jun 2016 15:08:48 +0200 Original-Received: by mail-it0-f70.google.com with SMTP id f6sf123210260ith.1 for ; Sun, 26 Jun 2016 06:08:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isocpp-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:message-id:in-reply-to:references:subject :mime-version:x-original-sender:reply-to:precedence:mailing-list :list-id:x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=wdxO+qdkFhoMFxyNw/hMBLz83xArGYo/uyrHirUOdDA=; b=z7/8SCKP9GlDMcdWMedJhBWBkLKPSx8mKRa6EnZWgMzoKOQ7E3Pdw29XBV7EIXNONV T4wrtEMoYoKierj0bZPllmR4GaGh8rp0Ju/1hKNhfjfzKaK3xFWaRrM+ZqvVaEqaonRb +uuBXJfZ2BgDWo4xKMkUz2A6zhe58XjZCzxvQwuRuS+fkR30rfCQ95+7ou6ltuu2OPq3 I77e4WGmhZxzGfZSzZFXS8zDWnylj8H9f+E9I975Ydkwhd1NZdZymIT9VTaXlqowhv7Y URtLx9eaWL2vcl98cKosOjTEl6M1Taaje8Jw+QDA6n0/nuGSS8ScybPV6b+2HuFbNWbM jngg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:message-id:in-reply-to:references:subject :mime-version:x-original-sender:reply-to:precedence:mailing-list :list-id:x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=wdxO+qdkFhoMFxyNw/hMBLz83xArGYo/uyrHirUOdDA=; b=f3fGWCZRVp4unyKmMkzgsfW4dCqTmdhkatDPgMsXZr0vxvGcoh3HAo19Kb504B3+xq X/3vGOMz8COTpmfKdCRqX9flMJQBFJPNfjuJHXGycvhkjF4en2vPW9dSNZkk04hHstEU 33B+T1KgaWM0I54N1lWijujLyBTfBhU4oFFnyj9jYQDX7kl4XP/XVZKTx2p1OYqk8zex zlxtYh/gqkAwvyODDp9qkfDi/OQB8sCk6+FCTz9gzEL8fje7pDIVnaMLCs498Di5P1m1 odggh3A8wDza1wbV5VHOuYmmR69ErAv2BPPQIyngFpIZs86XKn0wmEmsjeSRbNX34rR1 h+MQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:message-id:in-reply-to :references:subject:mime-version:x-original-sender:reply-to :precedence:mailing-list:list-id:x-spam-checked-in-group:list-post :list-help:list-archive:list-subscribe:list-unsubscribe; bh=wdxO+qdkFhoMFxyNw/hMBLz83xArGYo/uyrHirUOdDA=; b=PXoGvQNo1Luhllo5hmXKG/ccs8MrhKSXifgincGVP1XkCOp5xW2ZTT+2z8LVjgFK9G l3h3aSk8GVPWvO6nSTaRH0S0CQOcZzZorpd2VgD6nexl5YyyJDwdsAbuC9rVjTqPXb/U YeNDHU96qkWGydheA5LwZf+rlMBunFiyIhBlCdAlukubKP7CCbgJ/mmuEiq5hh2X1/AU 8J3V/EpTCZoKsGsg12JP2iQ0ErBfUoD8FN9mQyogIN0uidSi2uD14qjcPIP9nycOAkuz TtpwFKzTnN3ysH2r6SvOlBkNWjfMJP8WSJL12vEUVReZGmjFrLnWTSAwNoW9b51T8rQ1 R0zw== X-Gm-Message-State: ALyK8tJ6QjaI6cEYrhcgAg0mLWXyzH78obbKBrt1qvckO9FOJa6n6k9lNXhZcRVdRegrJw== X-Received: by 10.107.57.215 with SMTP id g206mr7354232ioa.8.1466946527179; Sun, 26 Jun 2016 06:08:47 -0700 (PDT) X-BeenThere: std-proposals@isocpp.org Original-Received: by 10.107.46.32 with SMTP id i32ls2078293ioo.30.gmail; Sun, 26 Jun 2016 06:08:45 -0700 (PDT) X-Received: by 10.36.17.211 with SMTP id 202mr146259itf.9.1466946525867; Sun, 26 Jun 2016 06:08:45 -0700 (PDT) In-Reply-To: <8ef0b8d3-3c57-4e54-8478-b81c2bd4207a@isocpp.org> X-Original-Sender: jmckesson@gmail.com Precedence: list Mailing-list: list std-proposals@isocpp.org; contact std-proposals+owners@isocpp.org List-ID: X-Spam-Checked-In-Group: std-proposals@isocpp.org X-Google-Group-Id: 399137483710 List-Post: , List-Help: , List-Archive: List-Subscribe: , List-Unsubscribe: , Xref: news.gmane.org gmane.comp.lang.c++.isocpp.proposals:26403 Archived-At: ------=_Part_163_695688364.1466946524785 Content-Type: multipart/alternative; boundary="----=_Part_164_1008336187.1466946524785" ------=_Part_164_1008336187.1466946524785 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Sunday, June 26, 2016 at 2:04:59 AM UTC-4, asor...@gmail.com wrote: > > =D0=B2=D0=BE=D1=81=D0=BA=D1=80=D0=B5=D1=81=D0=B5=D0=BD=D1=8C=D0=B5, 26 = =D0=B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., 8:52:16 UTC+3 =D0=BF=D0=BE=D0=BB=D1= =8C=D0=B7=D0=BE=D0=B2=D0=B0=D1=82=D0=B5=D0=BB=D1=8C Patrice Roy=20 > =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB: >> >> 18MB means we get away from many platforms where we are today. It's 9=20 >> times the size of my whole current project (with debug info in!) which r= uns=20 >> on embedded devices. Please remember that we want Unicode support, but n= ot=20 >> at the cost of not being able to support target platforms which are very= =20 >> much alive today. >> >> Unicode using codespace from 0 to 0x10FFFF. Therefore we need at most=20 > 0x110000 bits for single predicate function. 18 MB enough for hundred=20 > predicates. I'm don't think you really need so many. > Again, let's forget that most of that range is not actually assigned and=20 therefore takes up 0 bits. Not all of the properties in the Unicode tables are *binary*. Indeed, most= =20 are not. Case-conversion, for example, cannot be binary. It has to specify= =20 how you go from codepoint X to one or more codepoints YZW. For each=20 codepoint. That cannot take up a single bit per codepoint. That being said, I firmly believe that 18MB is much larger than the Unicode= =20 tables *need* to be. That there must be clever ways to make that table much= =20 smaller (on the order of hundreds of kilobytes rather than megabytes). But= =20 as of yet, I have not undertaken the task of *proving* that, so that=20 doesn't mean much. And even if I'm wrong, I bet we can provide certain very useful features=20 that require less than the full Unicode table space. For example, I'd bet= =20 that the non-compatibility Unicode normalization forms require much less=20 table space than the compatibility ones. I'd bet that grapheme cluster=20 iteration requires much less table space than case conversion. Again, not proven. But it'd certainly be a worthy research project. If=20 Unicode normalization functions only cost 15KB of executable room, that=20 might be a reasonable tradeoff. Obviously, if you don't call them at all,= =20 you should get zero increase. > --=20 You received this message because you are subscribed to the Google Groups "= ISO C++ Standard - Future Proposals" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to std-proposals+unsubscribe@isocpp.org. To post to this group, send email to std-proposals@isocpp.org. To view this discussion on the web visit https://groups.google.com/a/isocpp= ..org/d/msgid/std-proposals/4d975edd-1c9c-468d-b852-28e2a2108931%40isocpp.or= g. ------=_Part_164_1008336187.1466946524785 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On Sunday, June 26, 2016 at 2:04:59 AM UTC-4, asor...@gmai= l.com wrote:
= =D0=B2=D0=BE=D1=81=D0=BA=D1=80=D0=B5=D1=81=D0=B5=D0=BD=D1=8C=D0=B5, 26 =D0= =B8=D1=8E=D0=BD=D1=8F 2016 =D0=B3., 8:52:16 UTC+3 =D0=BF=D0=BE=D0=BB=D1=8C= =D0=B7=D0=BE=D0=B2=D0=B0=D1=82=D0=B5=D0=BB=D1=8C Patrice Roy =D0=BD=D0=B0= =D0=BF=D0=B8=D1=81=D0=B0=D0=BB:
18MB means we get away from many platforms where we are t= oday. It's 9 times the size of my whole current project (with debug inf= o in!) which runs on embedded devices. Please remember that we want Unicode= support, but not at the cost of not being able to support target platforms= which are very much alive today.

Unic= ode using codespace from 0 to 0x10FFFF. Therefore we need at most 0x110000 = bits for single predicate function. 18 MB enough for hundred predicates. I&= #39;m don't think you really need so many.
=

Again, let's forget that most of that range is not actually as= signed and therefore takes up 0 bits.

Not all of the properties in t= he Unicode tables are binary. Indeed, most are not. Case-conversion,= for example, cannot be binary. It has to specify how you go from codepoint= X to one or more codepoints YZW. For each codepoint. That cannot take up a= single bit per codepoint.

That being said, I firmly believe that 18= MB is much larger than the Unicode tables need to be. That there mus= t be clever ways to make that table much smaller (on the order of hundreds = of kilobytes rather than megabytes). But as of yet, I have not undertaken t= he task of proving that, so that doesn't mean much.

And e= ven if I'm wrong, I bet we can provide certain very useful features tha= t require less than the full Unicode table space. For example, I'd bet = that the non-compatibility Unicode normalization forms require much less ta= ble space than the compatibility ones. I'd bet that grapheme cluster it= eration requires much less table space than case conversion.

Again, = not proven. But it'd certainly be a worthy research project. If Unicode= normalization functions only cost 15KB of executable room, that might be a= reasonable tradeoff. Obviously, if you don't call them at all, you sho= uld get zero increase.

--
You received this message because you are subscribed to the Google Groups &= quot;ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to std-proposa= ls+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.= com/a/isocpp.org/d/msgid/std-proposals/4d975edd-1c9c-468d-b852-28e2a2108931= %40isocpp.org.
------=_Part_164_1008336187.1466946524785-- ------=_Part_163_695688364.1466946524785-- .