220 33310 <4d561e70-df04-4780-bae7-bdf1ae6b01f4@isocpp.org> article Path: news.gmane.org!.POSTED!not-for-mail From: Alexander Zaitsev Newsgroups: gmane.comp.lang.c++.isocpp.proposals Subject: Re: Memory mapped files and shared memory for C++ Date: Sat, 22 Jul 2017 10:09:55 -0700 (PDT) Lines: 393 Approved: news@gmane.org Message-ID: <4d561e70-df04-4780-bae7-bdf1ae6b01f4@isocpp.org> References: <7ee99206-74e9-4309-a5af-8974994698b3@isocpp.org> <1590671.jJ8zeR8SAH@tjmaciei-mobl1> Reply-To: std-proposals@isocpp.org NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_2262_64317658.1500743395285" X-Trace: blaine.gmane.org 1500743398 12440 195.159.176.226 (22 Jul 2017 17:09:58 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Sat, 22 Jul 2017 17:09:58 +0000 (UTC) To: ISO C++ Standard - Future Proposals Original-X-From: std-proposals+bncBDGLZGV44EEBBZENZ3FQKGQE244BP2I@isocpp.org Sat Jul 22 19:09:52 2017 Return-path: Envelope-to: gclcip-std-proposals@m.gmane.org Original-Received: from mail-qt0-f200.google.com ([209.85.216.200]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dYxuZ-0002rs-Vi for gclcip-std-proposals@m.gmane.org; Sat, 22 Jul 2017 19:09:52 +0200 Original-Received: by mail-qt0-f200.google.com with SMTP id u19sf27367888qtc.14 for ; Sat, 22 Jul 2017 10:09:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isocpp-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:message-id:in-reply-to:references:subject:mime-version :x-original-sender:reply-to:precedence:mailing-list:list-id :list-post:list-help:list-archive:list-subscribe:list-unsubscribe; bh=2XUnz1UMUlA4Cv5qAkijlfC/eaWFPkA9fZSM9z2ZXQo=; b=ajB7ZtlqNf5DiKhLgetD10vXPfAzYQO3PrJHtsMXhDlHnAdQ+vsHXtFTbTNoTYphMf u68b/rAIzYbeaLV3RKC6Uny1SK8ULbN3ksgL1urr26ZPDIK38tBqQGUmvxqA1VSfh/N+ wJ9rMClAek3igLrkfl5q8z9n3uQTiGUB6RCw07ykdBBfjEj5RFO2y3+o53QCQKXtwaKi V4vwr3qHezXc7W8xv7ZS5/zBusu3nS/ZEHnAx6zdUREjbgSFeHwkx36mXP7XcaclNNLY Uq9r9Q0ITGtxv9pNuw51x/Q+dKuVyytFR7BY3vJzi74CxO0FaXUeBoBMIsQ4PFy5ROuW 4RRQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:message-id:in-reply-to:references:subject:mime-version :x-original-sender:reply-to:precedence:mailing-list:list-id :list-post:list-help:list-archive:list-subscribe:list-unsubscribe; bh=2XUnz1UMUlA4Cv5qAkijlfC/eaWFPkA9fZSM9z2ZXQo=; b=lcjdQiqdHzYoYK3ThZSk8B1SqkYD89Z5Ub8RYI92zvb1qM+JgZ1aH+EFMJRDYozroC 8wY3BaA/OMQW+yyWbcSnCPRI0WIXD2Zf9qMTFl+Xg8wlUcwv6aCEOlbtKvz921ypLbwm Ov5fAotEOueLlKS2rS0mVnI1wFqih+WV06qzX3wPuUNIPKMN3ldg71dk9HiVtgAVyZ0s WvOM05AcZOMG3a3euiK9ZvcmS5TR9nbyyzU1vfCyZeBOEn3BChrm0ruvb2NLm0jHHH28 Qrevstkp2XSvYsF5q238WTVZJa7pMlUtLn0YqwtVdWVR8RSjFm4bXej4OjTEom4VONb1 mOwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:message-id:in-reply-to:references :subject:mime-version:x-original-sender:reply-to:precedence :mailing-list:list-id:x-spam-checked-in-group:list-post:list-help :list-archive:list-subscribe:list-unsubscribe; bh=2XUnz1UMUlA4Cv5qAkijlfC/eaWFPkA9fZSM9z2ZXQo=; b=kMWUCNIcjLy6AG+dYeawIToH7M4LgBPNzAlTC4C1/Dp3kzkSuaauLrUjVIeQVMBndg mdj8D2OFPLILE+95n30W29mh/DoAvehWwfQSO/GRIfjwdlH9azP/Z5kRis4rZaMjOO4n ppBJAVQYWmmsimKMSriOKEuezvP9QhblEKtC1FqvJmjepTmvURIvur2Q8dCreeU5Rtcw xF5AK+6c0KlFLeQPe8P7wMTaf8ts10EirDsu7uddphyohH5qs/qHBR4jmcXQEywZrFDn TOU4piNU3eaOI2WbPnjgXO4X9poVyMQHJmKpkLK6EyN9iNfb0TlRYQvzId90UgaA2nnA hwhw== X-Gm-Message-State: AIVw113/Qk/YQDQKB67p2+p+6dBeNdS3NCAZ3+MzP4dzMcYpl3W4phUh z8EH9EVixExztpUb X-Received: by 10.200.49.85 with SMTP id h21mr7241897qtb.115.1500743397286; Sat, 22 Jul 2017 10:09:57 -0700 (PDT) X-BeenThere: std-proposals@isocpp.org Original-Received: by 10.36.79.21 with SMTP id c21ls1178735itb.16.gmail; Sat, 22 Jul 2017 10:09:55 -0700 (PDT) X-Received: by 10.31.171.86 with SMTP id u83mr41954vke.19.1500743395838; Sat, 22 Jul 2017 10:09:55 -0700 (PDT) In-Reply-To: <1590671.jJ8zeR8SAH@tjmaciei-mobl1> X-Original-Sender: zamazan4ik@gmail.com Precedence: list Mailing-list: list std-proposals@isocpp.org; contact std-proposals+owners@isocpp.org List-ID: X-Google-Group-Id: 399137483710 List-Post: , List-Help: , List-Archive: List-Subscribe: , List-Unsubscribe: , Xref: news.gmane.org gmane.comp.lang.c++.isocpp.proposals:33310 Archived-At: ------=_Part_2262_64317658.1500743395285 Content-Type: multipart/alternative; boundary="----=_Part_2263_663891985.1500743395285" ------=_Part_2263_663891985.1500743395285 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thank you for a lot of suggestions. I will think about your comment. =D1=81=D1=83=D0=B1=D0=B1=D0=BE=D1=82=D0=B0, 22 =D0=B8=D1=8E=D0=BB=D1=8F 201= 7 =D0=B3., 4:28:48 UTC+3 =D0=BF=D0=BE=D0=BB=D1=8C=D0=B7=D0=BE=D0=B2=D0=B0= =D1=82=D0=B5=D0=BB=D1=8C Thiago Macieira=20 =D0=BD=D0=B0=D0=BF=D0=B8=D1=81=D0=B0=D0=BB: > > On Friday, 21 July 2017 16:26:20 PDT Alexander Zaitsev wrote:=20 > > Hello.=20 > >=20 > > In 2006 Ion Gazta=C3=B1aga (Boost.Interprocess and other great librarie= s=20 > author,=20 > > email: igaztanaga at gmail dot com) wrote proposal about memory mapped= =20 > > files and shared memory for C++ (Document number: N2044=3D06-0114). But= =20 > the=20 > > proposal is forgotten now.=20 > >=20 > > I want to reborn it, because i think it's very important thing for C++.= =20 > >=20 > > Please read the proposal and discuss about it.=20 > > Looks interesting, but falls short because it leaves extremely important= =20 > things out.=20 > > * needs an exception-free API=20 > > * Execute permissions are missing. Your classes support only as read-only= ,=20 > read-write or invalid. But memory mapped objects have more flags,=20 > especially=20 > the execute permission. In fact, the ability to *deny* execution is an=20 > important security feature.=20 > > * "copy on write" is probably the wrong name for the "private" property.= =20 > It's=20 > unintuitive that it would apply to a read-only region, but it does. It=20 > indicates whether you want to see changes made by other processes or not.= =20 > I=20 > advise changing the name back to "private" vs "shared".=20 > > * std::size_t may be the wrong type for the map size. It's valid for all= =20 > modern architectures, but in theory it just has to be wide enough to hold= =20 > the=20 > size of the biggest object. We may want instead to change the wording of= =20 > this=20 > type's definition (though note it's also in POSIX).=20 > > * Please update the address parameter to be nullptr, not 0, and update th= e=20 > get_address() documentation. Do all OSes guarantee they will map where=20 > asked?=20 > > (We can probably safely assume that mapping at nullptr is not supported b= y=20 > the=20 > C++ standard)=20 > > * For offset_t, you may want to use the filesystem API's offset type.=20 > > * For the flush() function, you need a way to explain whether it is neede= d=20 > or=20 > not. Most memory mappings don't need it.=20 > If you were thinking of msync(), then you're missing the arguments to tha= t=20 > function.=20 > > * I'm missing an equivalent of madvise() / posix_madvise()=20 > > * std::file_mapping needs a way to refer to an already-open file, without= =20 > needing to know its name (assuming it even has one). O_TMPFILE and memfd= =20 > files=20 > on Linux don't.=20 > > * what happens to the memory region if the std::file_mapping object is=20 > destroyed? On one hand, RAII principles would say the mapping is undone;= =20 > on=20 > another, that's not how the low-level OS works and it's often useful to= =20 > retain=20 > a mapping forever, without having to keep a (polymorphic) object around= =20 > that=20 > will never, ever be used again.=20 > > * std::shared_memory_object needs to specify which type of shared memory= =20 > it=20 > is. On Unix systems, we have two: POSIX (sys/mman.h, shm_open()) and=20 > System V=20 > (sys/shm.h, shmget()), with Linux having recently added a third (memfd).= =20 > They=20 > are completely disjoint namespaces and are identified differently.=20 > > * POSIX shared memory has the semantics of a file, including the access= =20 > permissions. Please merge with the filesystem API where they've looked=20 > into=20 > that. I suspect Windows ones are similar.=20 > > * the System V shared memory uses a key_t object for identification, not = a=20 > "const char *name". You can use a simple narrow-character string, but you= =20 > need=20 > to provide an "int proj_id" so it can be passed through ftok(3).=20 > > * just like path names, on Windows shared memory segments are not narrow= =20 > characters. You need to provide a wchar_t overload for Windows.=20 > > * Using truncate() for growing is weird. I know the POSIX API does that,= =20 > but=20 > bad prior art is not good reason. What did the filesystem API choose? If= =20 > given=20 > the option, I'd recommend "resize".=20 > > * Please write something about what happens if you map a size larger than= =20 > the=20 > object's size. Our experience in Qt is that some systems allow this,=20 > automatically growing the object that backs it, while others will just=20 > crash=20 > your application. It's ok to write that it's implementation-defined, but= =20 > you=20 > need to write it.=20 > > * for that matter, please update the API with the ability to seal memory-= =20 > mappable objects. See https://lwn.net/Articles/593918/,=20 > http://man7.org/linux/man-pages/man2/memfd_create.2.html. This was added= =20 > to=20 > Linux specifically to avoid DoS attacks where processes of different=20 > privileges=20 > are operating on the same shared memory object. Of course, implementation= -=20 > defined whether this is supported, but it's an important security feature= =20 > we=20 > should encourage application writers to use.=20 > > * you'll want to investigate cross-process thread signalling (semaphores)= =20 > too.=20 > Eventually, we'll need to discuss std::mutex on memory-mapped regions.=20 > > --=20 > Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org=20 > Software Architect - Intel Open Source Technology Center=20 > > --=20 You received this message because you are subscribed to the Google Groups "= ISO C++ Standard - Future Proposals" group. To unsubscribe from this group and stop receiving emails from it, send an e= mail to std-proposals+unsubscribe@isocpp.org. To post to this group, send email to std-proposals@isocpp.org. To view this discussion on the web visit https://groups.google.com/a/isocpp= ..org/d/msgid/std-proposals/4d561e70-df04-4780-bae7-bdf1ae6b01f4%40isocpp.or= g. ------=_Part_2263_663891985.1500743395285 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thank you for a lot of suggestions. I will think about you= r comment.

=D1=81=D1=83=D0=B1=D0=B1=D0=BE=D1=82=D0=B0, 22 =D0=B8=D1= =8E=D0=BB=D1=8F 2017 =D0=B3., 4:28:48 UTC+3 =D0=BF=D0=BE=D0=BB=D1=8C=D0=B7= =D0=BE=D0=B2=D0=B0=D1=82=D0=B5=D0=BB=D1=8C Thiago Macieira =D0=BD=D0=B0=D0= =BF=D0=B8=D1=81=D0=B0=D0=BB:
On= Friday, 21 July 2017 16:26:20 PDT Alexander Zaitsev wrote:
> Hello.
>=20
> In 2006 Ion Gazta=C3=B1aga (Boost.Interprocess and other great lib= raries author,
> email: igaztanaga at gmail dot com) wrote proposal about memory ma= pped
> files and shared memory for C++ (Document number: N2044=3D06-0114)= .. But the
> proposal is forgotten now.
>=20
> I want to reborn it, because i think it's very important thing= for C++.
>=20
> Please read the proposal and discuss about it.

Looks interesting, but falls short because it leaves extremely importan= t=20
things out.

* needs an exception-free API

* Execute permissions are missing. Your classes support only as read-on= ly,=20
read-write or invalid. But memory mapped objects have more flags, espec= ially=20
the execute permission. In fact, the ability to *deny* execution is an= =20
important security feature.

* "copy on write" is probably the wrong name for the "pr= ivate" property. It's=20
unintuitive that it would apply to a read-only region, but it does. It= =20
indicates whether you want to see changes made by other processes or no= t. I=20
advise changing the name back to "private" vs "shared&qu= ot;.

* std::size_t may be the wrong type for the map size. It's valid fo= r all=20
modern architectures, but in theory it just has to be wide enough to ho= ld the=20
size of the biggest object. We may want instead to change the wording o= f this=20
type's definition (though note it's also in POSIX).=20

* Please update the address parameter to be nullptr, not 0, and update = the=20
get_address() documentation. Do all OSes guarantee they will map where = asked?

(We can probably safely assume that mapping at nullptr is not supported= by the=20
C++ standard)

* For offset_t, you may want to use the filesystem API's offset typ= e.

* For the flush() function, you need a way to explain whether it is nee= ded or=20
not. Most memory mappings don't need it.
If you were thinking of msync(), then you're missing the arguments = to that=20
function.

* I'm missing an equivalent of madvise() / posix_madvise()

* std::file_mapping needs a way to refer to an already-open file, witho= ut=20
needing to know its name (assuming it even has one). O_TMPFILE and memf= d files=20
on Linux don't.

* what happens to the memory region if the std::file_mapping object is= =20
destroyed? On one hand, RAII principles would say the mapping is undone= ; on=20
another, that's not how the low-level OS works and it's often u= seful to retain=20
a mapping forever, without having to keep a (polymorphic) object around= that=20
will never, ever be used again.

* std::shared_memory_object needs to specify which type of shared memor= y it=20
is. On Unix systems, we have two: POSIX (sys/mman.h, shm_open()) and Sy= stem V=20
(sys/shm.h, shmget()), with Linux having recently added a third (memfd)= .. They=20
are completely disjoint namespaces and are identified differently.

* POSIX shared memory has the semantics of a file, including the access= =20
permissions. Please merge with the filesystem API where they've loo= ked into=20
that. I suspect Windows ones are similar.

* the System V shared memory uses a key_t object for identification, no= t a=20
"const char *name". You can use a simple narrow-character str= ing, but you need=20
to provide an "int proj_id" so it can be passed through ftok(= 3).

* just like path names, on Windows shared memory segments are not narro= w=20
characters. You need to provide a wchar_t overload for Windows.

* Using truncate() for growing is weird. I know the POSIX API does that= , but=20
bad prior art is not good reason. What did the filesystem API choose? I= f given=20
the option, I'd recommend "resize".

* Please write something about what happens if you map a size larger th= an the=20
object's size. Our experience in Qt is that some systems allow this= ,=20
automatically growing the object that backs it, while others will just = crash=20
your application. It's ok to write that it's implementation-def= ined, but you=20
need to write it.

* for that matter, please update the API with the ability to seal memor= y-
mappable objects. See https://lwn.net/Articles/593918/,=20
http://man7.org/linux/man-pages/man2/memfd_crea= te.2.html. This was added to=20
Linux specifically to avoid DoS attacks where processes of different pr= ivileges=20
are operating on the same shared memory object. Of course, implementati= on-
defined whether this is supported, but it's an important security f= eature we=20
should encourage application writers to use.

* you'll want to investigate cross-process thread signalling (semap= hores) too.=20
Eventually, we'll need to discuss std::mutex on memory-mapped regio= ns.

--=20
Thiago Macieira - thiago (AT) macieira.info - thiago (AT) kde.org
=C2=A0 =C2=A0Software Architect - Intel Open Source Technology Center

--
You received this message because you are subscribed to the Google Groups &= quot;ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to std-proposa= ls+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.= com/a/isocpp.org/d/msgid/std-proposals/4d561e70-df04-4780-bae7-bdf1ae6b01f4= %40isocpp.org.
------=_Part_2263_663891985.1500743395285-- ------=_Part_2262_64317658.1500743395285-- .