220 26414 article Path: news.gmane.org!not-for-mail From: Nicol Bolas Newsgroups: gmane.comp.lang.c++.isocpp.proposals Subject: Re: iswalpha and locales Date: Sun, 26 Jun 2016 12:53:01 -0700 (PDT) Lines: 169 Approved: news@gmane.org Message-ID: References: <5057e854-b2ea-4c39-81c7-367bc3e54080@isocpp.org> <8ef0b8d3-3c57-4e54-8478-b81c2bd4207a@isocpp.org> <4d975edd-1c9c-468d-b852-28e2a2108931@isocpp.org> <5188654.7iUmSHZqYX@tjmaciei-mobl1> Reply-To: std-proposals@isocpp.org NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_375_1600764210.1466970781683" X-Trace: ger.gmane.org 1466970785 13530 80.91.229.3 (26 Jun 2016 19:53:05 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sun, 26 Jun 2016 19:53:05 +0000 (UTC) To: ISO C++ Standard - Future Proposals Original-X-From: std-proposals+bncBCEKFTV6ZUMBBHXFYC5QKGQEY54A25Y@isocpp.org Sun Jun 26 21:53:05 2016 Return-path: Envelope-to: gclcip-std-proposals@m.gmane.org Original-Received: from mail-ob0-f197.google.com ([209.85.214.197]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1bHG76-00029i-Lk for gclcip-std-proposals@m.gmane.org; Sun, 26 Jun 2016 21:53:04 +0200 Original-Received: by mail-ob0-f197.google.com with SMTP id hx8sf292344738obb.0 for ; Sun, 26 Jun 2016 12:53:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isocpp-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:message-id:in-reply-to:references:subject:mime-version :x-original-sender:reply-to:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=c8H4MkNL1HpZyHB8BpFJQfZUVp8mVPnpm1gFgr8acKY=; b=WpCDXMNwssBlJfhqxzwQJC7csp4rhLLGYqd7UzOw/1w9TuEpDuZ8BwCyRsj6dSvRxY RQjztBN0X9XtEQN8gpGwBdrYj1N8L9Xon3JqgCUDG4E57N2SF9cRBLaU/yayLlKDL+IB LZqhOWKJHhXTkQRJ4AX0/d1E+7kUisssDH6qjFl2eZDRifH7fkNAZCndZgxuJnf58yMU YZWkcbnDzMhnxMkLvR0J28Cl6uTldyeC/VW6+XoRhdcBT4vicmDFHF8o/a3aLKaL1kzf 6L8AeTP+x36u5/EGW0aBNBE8vwEyYChVr0y5KqAbfpOLhu92cEBsOIzSp+BQeUCT98uE H/yg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:message-id:in-reply-to:references:subject:mime-version :x-original-sender:reply-to:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=c8H4MkNL1HpZyHB8BpFJQfZUVp8mVPnpm1gFgr8acKY=; b=Sm66rFcRHLlCvrheUBHru8d7PcMDmT0js9tqkRAmsNUD0+8dwhQGvAkClLV5MZzHTF Kuet/+/eRIsMcLJj18p02nvX43/cUSLsCRXedXPy54XCYECjIqncytPxTihekEcHoo12 xg7ntkp1QKlO6F8qfJ4UPcVTXkvk6wUM96GKuM4P9Gcman74OuPGWFdIt4pzuAM4lKUS Y9FeRImOuwxK2mTWqAMalSAxbfpiilWZlaKfBX71MgXY/0GoBu8Ffld1b6hwmbkZukgj 9IosB6VK65CenPjztPJ0IRtiil/hPvYyiX0fCHgsT6qjGPdEPlV5Xf4rFUaFcSOol4my GsZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:message-id:in-reply-to:references :subject:mime-version:x-original-sender:reply-to:precedence :mailing-list:list-id:x-spam-checked-in-group:list-post:list-help :list-archive:list-subscribe:list-unsubscribe; bh=c8H4MkNL1HpZyHB8BpFJQfZUVp8mVPnpm1gFgr8acKY=; b=liV9ha5XjmmyFD6OVGtGYj1P10TA8XHNZ3JtZBOgtLaLwGqV926bk7VssvtuawYGFa hGccDmkhYVUmIR5GB2RC1CNVEOl6n0TmARJgjpIwLXpUiJUKpsbPKubgjGZHqeuv0BLA gd5ocTb8/RyDvJpj6tQAi47ibSbydqU7a/tY3F0G7m7/o8WQmEDy+0Yfvwdbv8WmoGzd CCrJGVljssICrP3qY0F0Pbf4WyR/l0GlCWZD8V5wRmWfQBi4bCm7DkCxqdt8lt/1jX1l rXfHn8AlRlj/gg+OLQdMeYkqG1VxrG0OCs1lMXJgop2LJ3whFpjh11JOy/fVY05sDHab ULOw== X-Gm-Message-State: ALyK8tKQvB8oZ5EwBLjXr1pqglH+GlYtElM2yANnXJ1SyCSe8MKsezLmBemYoEuAW+7g9A== X-Received: by 10.157.43.17 with SMTP id o17mr1753415otb.36.1466970783626; Sun, 26 Jun 2016 12:53:03 -0700 (PDT) X-BeenThere: std-proposals@isocpp.org Original-Received: by 10.36.133.67 with SMTP id r64ls655222itd.38.canary; Sun, 26 Jun 2016 12:53:02 -0700 (PDT) X-Received: by 10.36.230.69 with SMTP id e66mr158299ith.0.1466970782712; Sun, 26 Jun 2016 12:53:02 -0700 (PDT) In-Reply-To: <5188654.7iUmSHZqYX@tjmaciei-mobl1> X-Original-Sender: jmckesson@gmail.com Precedence: list Mailing-list: list std-proposals@isocpp.org; contact std-proposals+owners@isocpp.org List-ID: X-Spam-Checked-In-Group: std-proposals@isocpp.org X-Google-Group-Id: 399137483710 List-Post: , List-Help: , List-Archive: List-Subscribe: , List-Unsubscribe: , Xref: news.gmane.org gmane.comp.lang.c++.isocpp.proposals:26414 Archived-At: ------=_Part_375_1600764210.1466970781683 Content-Type: multipart/alternative; boundary="----=_Part_376_1237769351.1466970781683" ------=_Part_376_1237769351.1466970781683 Content-Type: text/plain; charset=UTF-8 On Sunday, June 26, 2016 at 1:24:18 PM UTC-4, Thiago Macieira wrote: > > On domingo, 26 de junho de 2016 06:08:44 PDT Nicol Bolas wrote: > > That being said, I firmly believe that 18MB is much larger than the > Unicode > > tables *need* to be. That there must be clever ways to make that table > much > > smaller (on the order of hundreds of kilobytes rather than megabytes). > But > > as of yet, I have not undertaken the task of *proving* that, so that > > doesn't mean much. > > > > And even if I'm wrong, I bet we can provide certain very useful features > > that require less than the full Unicode table space. For example, I'd > bet > > that the non-compatibility Unicode normalization forms require much less > > table space than the compatibility ones. I'd bet that grapheme cluster > > iteration requires much less table space than case conversion. > > ICU comes with a tool to select which properties and which locales to > include > in your data pack. It's just not an easy tool to use and I personally know > of > no one that has successfully deployed the data file with it. > > Un-selecting entries from the "lines" from the database is often a short- > sighted decision. You may think "my application will not be run in > Thailand" > and thus remove support for Thai grapheme support along with its locale > information. But then you may get a Thai customer calling your > application's > support and they may not even be in Thailand. > > Un-selecting "columns" would be safer, but you often don't know which > properties your application needs. You might think like you said above > that > you don't need the non-compatibility normalisations, only to find out that > Internationalised Domain Names does need NFKC. > Well, removing specific properties is a compile-time decision, since those functions simply don't exist. Remember: we're not talking about what to "remove" necessarily; we're talking about what should be *added* to the standard library. And if we don't add the compatibility normalization forms (because we deem them to be too costly), then whatever IDN needs is irrelevant; the standard library simply doesn't support it. The general idea I'm trying to get to is that some operations are worth spending X amount of memory, and some operations are not. We should try to ascertain how much memory each Unicode operation that uses Unicode properties costs, so that we can determine which ones are worth supporting and which ones aren't. > Also, ICU 57.1 isn't 18 MB: > > $ v -h /usr/share/icu/57.1/icudt57l.dat > -rw-r--r-- 1 root root 25M Jun 15 06:14 /usr/share/icu/57.1/icudt57l.dat > They store it as a file to be directly included? No run-length encoding of series of elements that contain the same value or anything? -- You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group. To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org. To post to this group, send email to std-proposals@isocpp.org. To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/ecc0fc43-752f-42ba-9f9b-a47485324314%40isocpp.org. ------=_Part_376_1237769351.1466970781683 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
On Sunday, June 26, 2016 at 1:24:18 PM UTC-4, Thiago Macie= ira wrote:
On domingo, 26 de ju= nho de 2016 06:08:44 PDT Nicol Bolas wrote:
> That being said, I firmly believe that 18MB is much larger than th= e Unicode=20
> tables *need* to be. That there must be clever ways to make that t= able much
> smaller (on the order of hundreds of kilobytes rather than megabyt= es). But
> as of yet, I have not undertaken the task of *proving* that, so th= at
> doesn't mean much.
>
> And even if I'm wrong, I bet we can provide certain very usefu= l features
> that require less than the full Unicode table space. For example, = I'd bet
> that the non-compatibility Unicode normalization forms require muc= h less
> table space than the compatibility ones. I'd bet that grapheme= cluster
> iteration requires much less table space than case conversion.

ICU comes with a tool to select which properties and which locales to i= nclude=20
in your data pack. It's just not an easy tool to use and I personal= ly know of=20
no one that has successfully deployed the data file with it.

Un-selecting entries from the "lines" from the database is of= ten a short-
sighted decision. You may think "my application will not be run in= Thailand"=20
and thus remove support for Thai grapheme support along with its locale= =20
information. But then you may get a Thai customer calling your applicat= ion's =C2=A0
support and they may not even be in Thailand.

Un-selecting "columns" would be safer, but you often don'= t know which=20
properties your application needs. You might think like you said above = that=20
you don't need the non-compatibility normalisations, only to find o= ut that=20
Internationalised Domain Names does need NFKC.

Well, removing specific properties is a compile-t= ime decision, since those functions simply don't exist. Remember: we= 9;re not talking about what to "remove" necessarily; we're ta= lking about what should be added to the standard library. And if we = don't add the compatibility normalization forms (because we deem them t= o be too costly), then whatever IDN needs is irrelevant; the standard libra= ry simply doesn't support it.

The general idea I'm trying to= get to is that some operations are worth spending X amount of memory, and = some operations are not. We should try to ascertain how much memory each Un= icode operation that uses Unicode properties costs, so that we can determin= e which ones are worth supporting and which ones aren't.

<= blockquote class=3D"gmail_quote" style=3D"margin: 0;margin-left: 0.8ex;bord= er-left: 1px #ccc solid;padding-left: 1ex;">
Also, ICU 57.1 isn't 18 MB:

$ v -h /usr/share/icu/57.1/icudt57l.dat
-rw-r--r-- 1 root root 25M Jun 15 06:14 /usr/share/icu/57.1/icudt57l.dat

They store it as a file to be directly included? = No run-length encoding of series of elements that contain the same value or= anything?

--
You received this message because you are subscribed to the Google Groups &= quot;ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to std-proposa= ls+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.= com/a/isocpp.org/d/msgid/std-proposals/ecc0fc43-752f-42ba-9f9b-a47485324314= %40isocpp.org.
------=_Part_376_1237769351.1466970781683-- ------=_Part_375_1600764210.1466970781683-- .