220 24342 <9E5DE55D2546134DACFE4D3C117B36FD1F5A036F@us01wembx1.internal.synopsys.com> article Path: news.gmane.org!not-for-mail From: Tom Honermann Newsgroups: gmane.comp.lang.c++.isocpp.proposals Subject: Re: Text_view: A C++ concepts and range based character encoding and code point enumeration library Date: Sat, 13 Feb 2016 05:23:27 +0000 Lines: 105 Approved: news@gmane.org Message-ID: <9E5DE55D2546134DACFE4D3C117B36FD1F5A036F@us01wembx1.internal.synopsys.com> References: <9E5DE55D2546134DACFE4D3C117B36FD1F5955EB@us01wembx1.internal.synopsys.com> <5079622.tGPsnUbiJP@tjmaciei-mobl4> <9E5DE55D2546134DACFE4D3C117B36FD1F59AED4@us01wembx1.internal.synopsys.com> <2564403.1mZd4A4EWs@tjmaciei-mobl4> Reply-To: std-proposals@isocpp.org NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Trace: ger.gmane.org 1455341019 25780 80.91.229.3 (13 Feb 2016 05:23:39 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sat, 13 Feb 2016 05:23:39 +0000 (UTC) To: "std-proposals@isocpp.org" Original-X-From: std-proposals+bncBDT2VLVKUICRBUP37K2QKGQEWUGOZWA@isocpp.org Sat Feb 13 06:23:32 2016 Return-path: Envelope-to: gclcip-std-proposals@m.gmane.org Original-Received: from mail-ig0-f199.google.com ([209.85.213.199]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1aUSg7-0006fs-Rt for gclcip-std-proposals@m.gmane.org; Sat, 13 Feb 2016 06:23:32 +0100 Original-Received: by mail-ig0-f199.google.com with SMTP id sv7sf87065726igc.0 for ; Fri, 12 Feb 2016 21:23:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isocpp-org.20150623.gappssmtp.com; s=20150623; h=from:to:subject:thread-topic:thread-index:date:message-id :references:accept-language:content-language:content-type :mime-version:x-original-sender:x-original-authentication-results :reply-to:precedence:mailing-list:list-id:x-spam-checked-in-group :list-post:list-help:list-archive:list-subscribe:list-unsubscribe; bh=lHyH5/khgUfW3E+jTiRYRN9MsDaS5ZhfMqY+yUjTqvs=; b=ILgtJofVUbElulMOeYIE758/Kwz6oFP1T4XGbllkbleHejxNRhfnvEtZVysRqHgF8A N3rUVnZnZgzKXfEEbMK8EwB556J0HGFS9SoAcCmPdpuweLkTebjel0NCwjiw1V7zjQ49 AiFV4xcVN5imErXMEIiWND/sxAbuaj8Wr23VESyfGQAHsAu4D0m+3YlqO28tGy7R7XyL KOZ7a34Xs9Dnbu6nBADv0EmUs5TWTimyAZpTEjNZRf1V3pfvFtpiBxbfIiWRJTyoZ+H/ Cn2PeWS2T++XDyTREFC99IP5hN5JuMhJ6l1ZhI63aos1Ge5+4zA X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:to:subject:thread-topic:thread-index:date :message-id:references:accept-language:content-language:content-type :mime-version:x-original-sender:x-original-authentication-results :reply-to:precedence:mailing-list:list-id:x-spam-checked-in-group :list-post:list-help:list-archive:list-subscribe:list-unsubscribe; bh=lHyH5/khgUfW3E+jTiRYRN9MsDaS5ZhfMqY+yUjTqvs=; b=QQt1Z008UF9+5xrRzDf2ZkEEz68jXCkTMSYY10sVdXCUBT345WQGPILpq/TqQGOhAf pXqN6jB5hP4+1ITW3XDEIPfYncMO63zOSyCT03lbq0q+RT9yRwuYBb3qAFfsOVj9uuUb 9oasx+Z3c7zVyMozrhG+dShewGSyf7yPRpFYpWhN/w+zlRfmeDPimyiWwxixNXOzCV/h QbS6Pa1eV0X23ToKYFinl6sbboamCvi427/XBr0cFItPdeB6c3YNyQZnb7asyniEAg7s /dqAZeCfBwspAi6uWVRzKw43eadXsyTLKnwd+8UENsLn++x X-Gm-Message-State: AG10YOTyQGUwOFTAZqDXJ2pYAeAhREK23+byjb/3gzcfWfhi8B4ihDj/9pcJCxaYrFCgdQ== X-Received: by 10.50.79.161 with SMTP id k1mr1425059igx.5.1455341010919; Fri, 12 Feb 2016 21:23:30 -0800 (PST) X-BeenThere: std-proposals@isocpp.org Original-Received: by 10.50.30.166 with SMTP id t6ls24822igh.20.canary; Fri, 12 Feb 2016 21:23:29 -0800 (PST) X-Received: by 10.66.246.165 with SMTP id xx5mr7451065pac.87.1455341009512; Fri, 12 Feb 2016 21:23:29 -0800 (PST) Original-Received: from smtprelay.synopsys.com (us01smtprelay-2.synopsys.com. [198.182.47.9]) by mx.google.com with ESMTPS id y77si24928749pfa.187.2016.02.12.21.23.29 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 12 Feb 2016 21:23:29 -0800 (PST) Received-SPF: pass (google.com: best guess record for domain of Thomas.Honermann@synopsys.com designates 198.182.47.9 as permitted sender) client-ip=198.182.47.9; Original-Received: from dc8secmta2.synopsys.com (dc8secmta2.synopsys.com [10.13.218.202]) by smtprelay.synopsys.com (Postfix) with ESMTP id C190024E0528 for ; Fri, 12 Feb 2016 21:23:28 -0800 (PST) Original-Received: from dc8secmta2.internal.synopsys.com (dc8secmta2.internal.synopsys.com [127.0.0.1]) by dc8secmta2.internal.synopsys.com (Service) with ESMTP id ABF5EA4112 for ; Fri, 12 Feb 2016 21:23:28 -0800 (PST) Original-Received: from mailhost.synopsys.com (mailhost1.synopsys.com [10.12.238.239]) by dc8secmta2.internal.synopsys.com (Service) with ESMTP id 7BB5AA4102 for ; Fri, 12 Feb 2016 21:23:28 -0800 (PST) Original-Received: from mailhost.synopsys.com (localhost [127.0.0.1]) by mailhost.synopsys.com (Postfix) with ESMTP id 68E7BCF1 for ; Fri, 12 Feb 2016 21:23:28 -0800 (PST) Original-Received: from US01WEHTC2.internal.synopsys.com (us01wehtc2.internal.synopsys.com [10.12.239.237]) by mailhost.synopsys.com (Postfix) with ESMTP id 5BA44CF0 for ; Fri, 12 Feb 2016 21:23:28 -0800 (PST) Original-Received: from us01wembx1.internal.synopsys.com ([169.254.1.89]) by US01WEHTC2.internal.synopsys.com ([10.12.239.237]) with mapi id 14.03.0195.001; Fri, 12 Feb 2016 21:23:28 -0800 Thread-Topic: [std-proposals] Text_view: A C++ concepts and range based character encoding and code point enumeration library Thread-Index: AdFiOby8SrIQ2+guTYSef+0SA11AdQ== Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.13.184.19] X-Original-Sender: thomas.honermann@synopsys.com X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: best guess record for domain of Thomas.Honermann@synopsys.com designates 198.182.47.9 as permitted sender) smtp.mailfrom=Thomas.Honermann@synopsys.com Precedence: list Mailing-list: list std-proposals@isocpp.org; contact std-proposals+owners@isocpp.org List-ID: X-Spam-Checked-In-Group: std-proposals@isocpp.org X-Google-Group-Id: 399137483710 List-Post: , List-Help: , List-Archive: List-Subscribe: , List-Unsubscribe: , Xref: news.gmane.org gmane.comp.lang.c++.isocpp.proposals:24342 Archived-At: On 2/10/2016 1:47 PM, Thiago Macieira wrote: > On quarta-feira, 10 de fevereiro de 2016 08:11:56 PST Tom Honermann wrote: >>> The error checking above is that next() silently replaces an invalid >>> decoding with the replacement character (U+FFFD). I suppose you could use >>> std::expected for your code. >> >> As I mentioned in my reply to Mathias, at one point I was working on a >> design that would allow this behavior to be configurable. It still >> isn't clear to me how beneficial that flexibility would be. > > Well, for those of us who will not touch exceptions with a 6-foot pole, your > code changes from "useless academic toy" to "useful framework". Understood. I didn't intended for exceptions to be the only error handling mechanism; I just haven't gotten around to doing something about it yet. > Using std::expected allows to have the best of all worlds: > > - want to have exceptions? Just use the value and it will throw if it's in the > wrong state > - want to have a silent replacement? .value_or(0xfffd) > - want to check errors without exceptions? there's a function Fitting std::expected into the low-level encoding and decoding interfaces should be straight forward. I don't see a way to utilize it within the iterator interface though. Using std::expected as the value_type of an iterator would be ... interesting at best. I've been thinking of supporting something like the following (perhaps made more general, with member functions that receive the relevant underlying code unit iterators and can return substitution characters or throw custom exceptions). struct my_text_view_policy { static const X on_underflow = save_state; // or throw_exception static const X on_invalid = subst_replacement; // or throw_exception }; std::string s = ...; auto tv = make_text_view(s); Thoughts on this approach appreciated. >>> ifstream is never a good reference for me. The only good thing about >>> iostreams for me are cout and cerr. cin, fstream, stringstream, etc., are >>> overkills and complex, so they never enter my projects. >> >> Fair enough, but I think the point stands that this kind of scenario can >> be addressed by a lower level buffered iterator abstraction. > > That would be buffering over my buffer. That's more memory allocated and > possibly introducing delays like networking bufferbloat (see > ) The solution I'm envisioning wouldn't entail double buffering. I'm just suggesting that the buffer management be left to the underlying code unit iterators. I've implemented a solution in the past that used a sliding window buffer. >> The only concern that I have about the above is that it leaves open the >> possibility for trailing code units (e.g., garbage at the end of the >> encoded text) to go unnoticed. In a non-buffering scenario, an iterator >> might silently compare to end even though there are code units >> remaining. The developer might care about these, or they might not. > > Indeed. That reminds me of the qstring.cpp function convertCase. Before > looping over the actual data, it does: > > // this avoids out of bounds check in the loop > while (e != p && e[-1].isHighSurrogate()) > --e; > > This probably means that iterators are the wrong tool for this job, at least > the way that the Standard Library understands iterators to be. That may be, though I think the iterator approach will remain useful for many use cases. For other cases, coding to the lower level interfaces is an option. > We've already talked about how they are stateful and know about the end > position. Now we need to be sure that they are properly disposed of, by > checking the saved state after the last character. > > Please also add to your calculations the fact that you'll need to have ucnv or > iconv_t objects behind the scenes. There's a whole side of object management. > > This is really a class with an invariant to be protected, not an iterator. Assuming implementation is provided by either ICU or iconv, yes. Iterators are allowed to reference the range object from which they were created, so management can be consolidated there. For views, this would imply additional shared resource management, perhaps managed via std::shared_ptr. Tom. -- --- You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group. To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org. To post to this group, send email to std-proposals@isocpp.org. Visit this group at https://groups.google.com/a/isocpp.org/group/std-proposals/. .