220 30440 <07f6e085-f75b-41f0-b42d-8a2704dbdb9a@isocpp.org> article Path: news.gmane.org!.POSTED!not-for-mail From: Nicol Bolas Newsgroups: gmane.comp.lang.c++.isocpp.proposals Subject: Re: string/vector extensions for C interop Date: Mon, 9 Jan 2017 17:35:54 -0800 (PST) Lines: 363 Approved: news@gmane.org Message-ID: <07f6e085-f75b-41f0-b42d-8a2704dbdb9a@isocpp.org> References: <06264fe0-3983-4128-aa91-76b5d128b385@isocpp.org> Reply-To: std-proposals@isocpp.org NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_1322_1747381655.1484012154749" X-Trace: blaine.gmane.org 1484012172 15093 195.159.176.226 (10 Jan 2017 01:36:12 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Tue, 10 Jan 2017 01:36:12 +0000 (UTC) Cc: gmisocpp@gmail.com To: ISO C++ Standard - Future Proposals Original-X-From: std-proposals+bncBCEKFTV6ZUMBB67U2DBQKGQECXLWKRY@isocpp.org Tue Jan 10 02:36:01 2017 Return-path: Envelope-to: gclcip-std-proposals@m.gmane.org Original-Received: from mail-io0-f199.google.com ([209.85.223.199]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cQlLs-0001tt-Cd for gclcip-std-proposals@m.gmane.org; Tue, 10 Jan 2017 02:35:52 +0100 Original-Received: by mail-io0-f199.google.com with SMTP id 67sf81843415ioh.1 for ; Mon, 09 Jan 2017 17:35:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isocpp-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:message-id:in-reply-to:references:subject :mime-version:x-original-sender:reply-to:precedence:mailing-list :list-id:x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=sXrUCwBC+UlWwVpU/Jepv7CBgVEYpVj0ch/QTMv+rQ8=; b=PAoR/SgzY1a3wH9UTLugBgDs3KynqBkGGCXev+Ar78Y6S/v2UiArG/ENUU9NO2z3HP wNM8QdSeqgDYql7kEUw4ci89ojgQ0q14JiTAi7qG1VExqfVjBOwQMMOXMiwGmxuTc91K HFrXvxj1neaTKuIKH0HFBpR+Ps55WUeGXurVTlNvAG5gsqW9Xwrw3OGTkkU1PHPuf2zG r6fPrYJMkeSi9EYABLun9kwcf7WsFu0K7YITIDhIeAcLxjYmOp3AE1LyRgkS1/mSubVG 9mir8LSMgaThNrGLN9xc0yLgNy57+XSC4R21mNsaSzpG+YXyDI+NI2TjpKN9MX4oFVSg bYww== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:message-id:in-reply-to:references:subject :mime-version:x-original-sender:reply-to:precedence:mailing-list :list-id:x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=sXrUCwBC+UlWwVpU/Jepv7CBgVEYpVj0ch/QTMv+rQ8=; b=MPDa+GP7OWvDIKHBaAeKh/FkPsX3ZcmHOLA/Y1+0t1BM8MlyJAgXusupGCvFs1bzXm hyq7TVWAkYEpNS3qRDjHbpeN/QtBrnfD0rqrU9aHeNgMLXqwuy53GdlDUyO184Hagdiw JW/jDKTBQUhl65wMHo0z/xyiZcXGUqQpeCygp/N7C35J6I4FgN5A3GENUJ1TgTtaaQ6u 8wQCHLfHNJdp/4KU79xkiLh/WXwrpus4oQa4FtR97faWsdHAFH9G1waVJISxiDd5+p/W baM4GN3sQCqHBuVi7l1haWcvxS/47EtNJp1FoAzEVfuQ4toTRk/JPNuwwA457or0uTLU oBXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:message-id:in-reply-to :references:subject:mime-version:x-original-sender:reply-to :precedence:mailing-list:list-id:x-spam-checked-in-group:list-post :list-help:list-archive:list-subscribe:list-unsubscribe; bh=sXrUCwBC+UlWwVpU/Jepv7CBgVEYpVj0ch/QTMv+rQ8=; b=rweQEZ258OldGsLFNb6c5i519cuhJywk29NIni8yT30UtvHcu1OoO52SVZ/OvLRBsn MFZ/Z63/h5QX7NAyfVB2K712hscZsouee1nCVbs2U1XX+Dxc3z3iDMGnYbvZBqLS7/lI o6ddBQfhevFfAuR9ZbHhbEOYIEueKn3KH9cHNyAW3HjlmrtjGxrqj1DJzbGAvv+hglTA +BJfM7K6ggMv8MhLx3dv/o5xCK87aL30q3SHCXyesKh7Fki8TiOmhlZUzoEccKtIZLIW 309Uas0GDDwLBe9fAVdABPKrFHxWIfPGzgF9+O2m60ZuFF4Z54WYzAiJ/Nh/k446hCnl 47lg== X-Gm-Message-State: AIkVDXKxmM0XRRtF/qRyfr/j6o8DNQzadNvJhLuUobBugRi3AO9fNdCvFA25uXfc5zdGBw== X-Received: by 10.107.40.2 with SMTP id o2mr349831ioo.1.1484012156102; Mon, 09 Jan 2017 17:35:56 -0800 (PST) X-BeenThere: std-proposals@isocpp.org Original-Received: by 10.157.3.235 with SMTP id f98ls201633otf.16.gmail; Mon, 09 Jan 2017 17:35:55 -0800 (PST) X-Received: by 10.157.12.165 with SMTP id b34mr18266otb.2.1484012155268; Mon, 09 Jan 2017 17:35:55 -0800 (PST) In-Reply-To: <06264fe0-3983-4128-aa91-76b5d128b385@isocpp.org> X-Original-Sender: jmckesson@gmail.com Precedence: list Mailing-list: list std-proposals@isocpp.org; contact std-proposals+owners@isocpp.org List-ID: X-Google-Group-Id: 399137483710 List-Post: , List-Help: , List-Archive: List-Subscribe: , List-Unsubscribe: , Xref: news.gmane.org gmane.comp.lang.c++.isocpp.proposals:30440 Archived-At: ------=_Part_1322_1747381655.1484012154749 Content-Type: multipart/alternative; boundary="----=_Part_1323_478382746.1484012154749" ------=_Part_1323_478382746.1484012154749 Content-Type: text/plain; charset=UTF-8 On Monday, January 9, 2017 at 7:48:48 PM UTC-5, gmis...@gmail.com wrote: > > string/vector extensions for C interop > > I've started this thread so the original thread it came doesn't continue > to get re-directed. > I've taken some comments from that to restart the discussion and re-worked > my example a bit. So: > > We seem to have a situation where C is in free-fall decline (at least on > the tiobe index). > > But if we can do more to support C programmers moving over to C++ it would > be a win for both camps. > C++ users could use certain performance tuning in this area regardless > because we use C. > > To quote r very own mr smith: > > "The fact that you can't resize a string or vector without initializing > its elements is a measurable and significant performance problem for some > applications; I have seen people invent their own string type or do truly > horrible things to std::string (directly poking at its internals) to > circumvent that." > > So what can we do to stop this? > > > I have seen a lot of code like this that I'd like to not write/fix/use: > > char* do_something_for_c_library() > { > std::string s; > s.resize(n+1); // We don't want zero fill here. > // SomeCAPI takes a pointer an a length and returns the actual length it > filled. > auto len = SomeCAPI(s.data(), n); /* Win32 I'm looking at you. */ > s.resize(len) > > s += "hello"; > > char* ptr = malloc(s.length()+1); > if (!ptr) > return nullptr; > > strcpy(ptr, s.data()); > > return ptr; > } > > > There are various reasons people use string like this (but vector also). > These are: > 1. They just intend to return a string eventually but need data from a C > like API. Fill can be expensive. > 2. They need some RAII thing so if there's an exception the destructor > will release the memory. > 3. They could use unique_ptr but they want to do some string'y things > first. > 4. They want string for the small buffer optimization. > 5. They sometimes need to return the buffer to C. > > It doesn't seem like a big deal, but this is often in performance critical > code. The sizes aren't always trivial. > So why can't we make this easier and more efficient? > > > To get this discussion started, how about: > > char* do_something_for_c_library() > { > std::string s; > > auto len = SomeCAPI(s.uinitialized_data_at(0, n+1), n); > s.resize(len); > > s+="hello"; > return s.detach_and_string_terminate(); // And provide basic detach also. > } > > s.uinitialized_data_at does this: > Ensures that a range of elements starting at 0 for length n exists. > and if it doesn't it extends it without initialization. > The starting position should exist or it's an error unless it is 0 and the > string is empty. > if it can't allocate or the offset or length is bad the it throws a > std::logic_error(); explaining the problem. > > > I don't see why we can't provide this same API for vector. > detach_and_string_terminate() could just be enable_if'd only types of > char/wchar_t/unsigned char/char32_t/char16_t. > > > unintialized_data_at() could just be enabled for these types too for now > if that helps safety. > I think this would: > 1. Increase performance as it enables a lot less pointless initializing of > data that will only be overwritten. > 2. Increase performance because it enables a lot of less excess > allocations in many cases. > 3. Reduce code size because the library is doing the work. > 4. Improve safety because the library is doing the safety checking for the > main fail points that users often don't check. > It has to do some checking anyway because it may be re-allocating. > 5. Help C programmers and C++ programmers who have to use or enable C > api's. > > We could provide data_at() too. These can be free functions perhaps. > I would like an API to request allocation of an exact buffer size too but > that doesn't have to be these API's. > > The detach api proposal can be separate from the data_at idea. > I don't think this idea needs to wait for some perfect language extensions. > > Questions: > Why not basically this? > Several reasons. First, `basic_string` is intended to be able to have small string optimization (SSO). This means that strings below a certain size are stored internally rather than allocating memory. Thus, your "detach" function doesn't "detach" something in all cases. Second, `basic_string` and `vector` both allocate memory based on provided allocators. So... how will the user destroy it? They cannot call `delete[]` on it, since it was not allocated with `new[]`. The only way to destroy this object us to use the same allocator that was used to allocate it, or a compatible one. Not only that, the person receiving the pointer has no idea how many objects are in the array, so unless `T` has a trivial destructor, you need to know how many items to destroy. And you need to remember to destroy them in *reverse* order. Third, your APIs are really bad. Inserting default-initialized objects shouldn't use radically different APIs from inserting value-initialized objects. Your API also assumes that `T` is a type which can tolerate being uninitialized. We shouldn't make APIs that break the C++ object model; we should make them that actually work with that model. The goal is this: T *t = new T[256]; vector tv(256, ...); Both of these arrays should perform the same amount of initializing of their respective arrays. If `T` is trivially default constructible, then both of these will be uninitialized. If `T` has a non-trivial default constructor, then it will be called 256 times in both cases. Because by the rules of C++, that's what `T` requires in order to be a live object. The goal should *not* be to have `vector/string` emulate `malloc`/casting/etc. If you want to violate the C++ object model, that's your business, but we shouldn't have standard library types let you do that. Do we really need to wait 3 years for ALL of this? > Yes. Be glad it's just 3 years. > But is providing the minimum to support a minimum of the > uninitialized_data_at routine so hard that we need to wait? > You cannot get something into the standard right now just because you really want it. C++17 was feature-complete *months ago*. The ship has sailed; it's not coming back into port. The next ship launches in 3 years. Get ready for it. If there's something better. What is it? > Well, there's what I already suggested. I also spent a little time thinking about a memory detachment API for vector, one that would actually recognize things like the fact that allocators exist ;) Since then, we've had the introduction of `map`/`set` extract/merge functions, and we see an alternate way to handle transfer of such objects. The design of such an API should mirror them, not deal in direct pointers to something. Also, such a design should include the ability to hand the system existing memory (and an allocator for deleting it), which can then be transferred into a `vector`/`string`. -- You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group. To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org. To post to this group, send email to std-proposals@isocpp.org. To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/07f6e085-f75b-41f0-b42d-8a2704dbdb9a%40isocpp.org. ------=_Part_1323_478382746.1484012154749 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable


On Monday, January 9, 2017 at 7:48:48 PM UTC-5, gm= is...@gmail.com wrote:
string/vector extensions for C interop

I've started this thread so the original thread it came=C2=A0doesn&#= 39;t continue to get=C2=A0re-directed.
I've taken some commen= ts from that to restart the discussion and re-worked my example a bit. So:<= /div>

We seem to have a situation where C is in free-fall decline (= at least on the tiobe index).

But if we can do more to support C pr= ogrammers moving over to C++ it would be a win for both camps.

C++ = users=C2=A0could use certain performance tuning in this area regardless bec= ause we use C.

To quote r very own mr smith:

"Th= e fact that you can't resize a string or vector without initializing it= s elements is a measurable and significant performance problem for some app= lications; I have seen people invent their own string type or do truly horr= ible things to std::string (directly poking at its internals) to circumvent= that."


So what can we do to stop this?

I have seen a lot of code like this that I'd like to not write/fix= /use:


char* do_something_for_c_library()
{
=C2=A0std::str= ing s;
=C2=A0s.resize(n+1); // We don't want zero fill here.
// SomeCAPI takes a pointer an a length and returns the actual length it= filled.
=C2=A0auto len =3D SomeCAPI(s.data(), n); /* Win32 I'm look= ing at you. */
=C2=A0s.resize(len)

=C2=A0s +=3D "hello"= ;;

=C2=A0char* ptr =3D malloc(s.length()+1);
=C2=A0if (!ptr)
= =C2=A0=C2=A0return nullptr;

=C2=A0strcpy(ptr, s.data());

= =C2=A0return ptr;
}


There are various reasons people use strin= g like this (but vector also). These are:
1. They just intend to return = a string eventually but need data from a C like API.=C2=A0Fill can be expen= sive.
2. They=C2=A0need some RAII thing so if there's an exception t= he destructor will release the memory.
3. They could use unique_ptr but = they want to do some string'y things first.
4. They want string for= =C2=A0the small buffer optimization.
5. They sometimes need to return th= e buffer to C.


It doesn't seem like a big deal, = but this is often in performance critical code. The sizes aren't always= trivial.
So why can't we make this easier and more efficient= ?


To get this discussion started,= =C2=A0how about:

char* do_something_for_c_library()<= br>{
=C2=A0std::string s;

=C2=A0auto len =3D SomeCAPI(s.uinitializ= ed_data_at(0, n+1), n);
=C2=A0s.resize(len);

=C2=A0s+=3D"= ;hello";
=C2=A0return s.detach_and_string_terminate(); // And = provide basic detach also.
}

s.uinitialized_data_at does this:
= Ensures that a range of elements starting at 0 for length n exists.
and = if it doesn't it extends it without initialization.
The starting pos= ition should exist or it's an error unless it is 0 and the string is em= pty.
if it can't allocate or the offset or length is bad the it thro= ws a std::logic_error(); explaining the problem.


I don't see = why we can't provide this same API for vector.
detach_and_string_ter= minate() could just be enable_if'd only types of char/wchar_t/unsigned = char/char32_t/char16_t.


unintialized_data_at() could just be enab= led for these types too for now if that helps safety.

I think this = would:
1. Increase performance as it enables a lot less pointless initia= lizing of data that will only be overwritten.
2. Increase performance be= cause it enables a lot of less excess allocations in many cases.
3. Redu= ce code size because the library is doing the work.
4. Improve sa= fety because the library is doing the safety checking for the main fail poi= nts that users often don't check.
It has to do some checking = anyway=C2=A0because it may be re-allocating.
5. Help C programmers and C= ++ programmers who have to use or enable C api's.

<= div>We could provide data_at() too. These can be free functions perhaps.
I would like an API to request allocation of an exact buffer size t= oo but that doesn't have to be these API's.

The detach api proposal can be separate from the data_at idea.
= I don't think this idea needs to wait for some perfect language extensi= ons.

Questions:
Why not basically this?<= /div>

Several reasons.

First, `basic_stri= ng` is intended to be able to have small string optimization (SSO). This me= ans that strings below a certain size are stored internally rather than all= ocating memory. Thus, your "detach" function doesn't "de= tach" something in all cases.

Second, `basic_string` and `vecto= r` both allocate memory based on provided allocators. So... how will the us= er destroy it? They cannot call `delete[]` on it, since it was not allocate= d with `new[]`. The only way to destroy this object us to use the same allo= cator that was used to allocate it, or a compatible one. Not only that, the= person receiving the pointer has no idea how many objects are in the array= , so unless `T` has a trivial destructor, you need to know how many items t= o destroy. And you need to remember to destroy them in reverse order= ..

Third, your APIs are really bad. Inserting default-initialized obj= ects shouldn't use radically different APIs from inserting value-initia= lized objects. Your API also assumes that `T` is a type which can tolerate = being uninitialized. We shouldn't make APIs that break the C++ object m= odel; we should make them that actually work with that model.

The go= al is this:

T *t =3D new T[256];
vector<T> tv(256, ...);

Both of these arrays s= hould perform the same amount of initializing of their respective arrays. I= f `T` is trivially default constructible, then both of these will be uninit= ialized.=C2=A0 If `T` has a non-trivial default constructor, then it will b= e called 256 times in both cases. Because by the rules of C++, that's w= hat `T` requires in order to be a live object.

The goal should no= t be to have `vector/string` emulate `malloc`/casting/etc. If you want = to violate the C++ object model, that's your business, but we shouldn&#= 39;t have standard library types let you do that.

Do we really need to= wait 3 years for ALL of this?

Yes. Be gla= d it's just 3 years.
=C2=A0
But is=C2=A0providing=C2=A0the min= imum to support a minimum of the uninitialized_data_at=C2=A0routine=C2=A0so hard that we need to wait?

You= cannot get something into the standard right now just because you really w= ant it. C++17 was feature-complete months ago. The ship has sailed; = it's not coming back into port.

The next ship launches in 3 year= s. Get ready for it.

If there's something better. What is it?

Well, there's what I already suggest= ed.

I also spent a little time thinking about a memory detachment AP= I for vector, one that would actually recognize things like the fact that a= llocators exist ;) Since then, we've had the introduction of `map`/`set= ` extract/merge functions, and we see an alternate way to handle transfer o= f such objects.

The design of such an API should mirror them, not de= al in direct pointers to something. Also, such a design should include the = ability to hand the system existing memory (and an allocator for deleting i= t), which can then be transferred into a `vector`/`string`.

--
You received this message because you are subscribed to the Google Groups &= quot;ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to std-proposa= ls+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.= com/a/isocpp.org/d/msgid/std-proposals/07f6e085-f75b-41f0-b42d-8a2704dbdb9a= %40isocpp.org.
------=_Part_1323_478382746.1484012154749-- ------=_Part_1322_1747381655.1484012154749-- .