220 24813 article Path: news.gmane.org!not-for-mail From: Sean Middleditch Newsgroups: gmane.comp.lang.c++.isocpp.proposals Subject: Re: Modules are hard for build tools Date: Sun, 28 Feb 2016 17:41:06 -0800 (PST) Lines: 199 Approved: news@gmane.org Message-ID: References: Reply-To: std-proposals@isocpp.org NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_101_638399159.1456710066362" X-Trace: ger.gmane.org 1456710075 21136 80.91.229.3 (29 Feb 2016 01:41:15 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 29 Feb 2016 01:41:15 +0000 (UTC) To: ISO C++ Standard - Future Proposals Original-X-From: std-proposals+bncBCDODCNR2QPRBM6DZ23AKGQE2RQGKZA@isocpp.org Mon Feb 29 02:41:09 2016 Return-path: Envelope-to: gclcip-std-proposals@m.gmane.org Original-Received: from mail-yk0-f198.google.com ([209.85.160.198]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1aaCph-00065N-5E for gclcip-std-proposals@m.gmane.org; Mon, 29 Feb 2016 02:41:09 +0100 Original-Received: by mail-yk0-f198.google.com with SMTP id u9sf226135585ykd.3 for ; Sun, 28 Feb 2016 17:41:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isocpp-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:message-id:in-reply-to:references:subject:mime-version :x-original-sender:reply-to:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=KlbhG/foQFtXWaOGgqAGxU+JSFezxFLbC/mXWq34jr4=; b=nZdQXlZHG26P7WHWrvmvn92HoXWrmVn1xbx/eRhPHB6w9YPIYvg6EDVEVitxvYNbPv zdLIlxXEiOinpP4U1be1O5AQ0YYjb+6ygFmGTCVzTeL7HYtQpKHP4+GP94WjuEkH1P0s 9hzsWezKTkCR0hQURrm7RgDTKel/W7QthbPbR5ciLZTcmdTUQAxL7ElKEX2DyXBoDlFV f0pCe3FEAD1OYf0FGeS8hALblwRGXCl2FTAwwQAUZ2sP0NA6iK5AezJS8Q0tBE4hFE3i 8JgZB+vRQ6iNzyRwjcThBnuFmPrbShp/0UvVYOxy8AEXrtjDVvUdEaD3o7/suEebnGSV PXkA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:message-id:in-reply-to:references:subject:mime-version :x-original-sender:reply-to:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=KlbhG/foQFtXWaOGgqAGxU+JSFezxFLbC/mXWq34jr4=; b=t+YR05vQZwSZvQ92+R3j1HvUNNqiRJbh9k7S8LYX1EtN6gOT4ExdK7fmKgrmwMo8bv Mx2HWjC/NKplkD/g3p1whHAVwDUM5hstQcvp2LCaBhR/MOrXnGF4l/Ecx7GmNaDlXOKp 5QF4u1/0pyIImzjk5HU78g+7ilkptCPg8FywWvrIfGNwYETVrtRci+eI4VHYu1FXwzRE DCdklAv5oPa61h40yIhXlaswuXDmr7D2YODAqHw7KLogLys+eGXALwcTbLPHlA2pDoCQ SW4qaZ/pQWFrtn9oU2rG/rXnnE44icTkL4qxg075GOgqUk5GPze/PE+wz78WkeUK0Yo7 /srg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:message-id:in-reply-to:references :subject:mime-version:x-original-sender:reply-to:precedence :mailing-list:list-id:x-spam-checked-in-group:list-post:list-help :list-archive:list-subscribe:list-unsubscribe; bh=KlbhG/foQFtXWaOGgqAGxU+JSFezxFLbC/mXWq34jr4=; b=mUZOjQdoL3vPVI7qH/2cnL8F5YH/i3Rpmjg7S0WaAq8TlDKqwZIqnZI3vJcYlzOKqA I3Z5HVEFaZK6O4ok227DORd810agAY5Z+lHRlOdrh46neuQAHKKYXny07oMUBm+1Mq3l MjAbSzjLFHwqnIQp+sCqn1u4MHGHxYsUbCno7OxYBbAVIEfCrRnIYnUBeK1Rmb/YEfDG BvmYuRQ2cgrXuT51zDIU50ytcLt1fkkdcXXXHxQyLlthjypLQgZrGeKBZBCWOr+ogQuZ V86IWodoj28v447FXJzSrEF5cXu0lMESUn+BTVr55DYfzHneKJ5hzfDKtrjtMI09rfVh 5igQ== X-Gm-Message-State: AD7BkJKieEA6jYGzSxV6RaSS0Dz6lD2sMhvXybyJnPdiQoi5NgeDszNZwq3UbVxp8KDI8g== X-Received: by 10.129.146.73 with SMTP id j70mr10807991ywg.25.1456710068040; Sun, 28 Feb 2016 17:41:08 -0800 (PST) X-BeenThere: std-proposals@isocpp.org Original-Received: by 10.107.164.234 with SMTP id d103ls1259032ioj.74.gmail; Sun, 28 Feb 2016 17:41:06 -0800 (PST) X-Received: by 10.50.103.68 with SMTP id fu4mr123029igb.9.1456710066927; Sun, 28 Feb 2016 17:41:06 -0800 (PST) In-Reply-To: X-Original-Sender: Sean.Middleditch@gmail.com Precedence: list Mailing-list: list std-proposals@isocpp.org; contact std-proposals+owners@isocpp.org List-ID: X-Spam-Checked-In-Group: std-proposals@isocpp.org X-Google-Group-Id: 399137483710 List-Post: , List-Help: , List-Archive: List-Subscribe: , List-Unsubscribe: , Xref: news.gmane.org gmane.comp.lang.c++.isocpp.proposals:24813 Archived-At: ------=_Part_101_638399159.1456710066362 Content-Type: multipart/alternative; boundary="----=_Part_102_513709644.1456710066363" ------=_Part_102_513709644.1456710066363 Content-Type: text/plain; charset=UTF-8 I'll start off by saying that I don't think that Abyx's worries are a significant problem. Yes, build systems will need to change, but they've needed to change for 20 years now. If modules are the impetus for the community to finally make building C++ not ridiculous, that's just fine with me. I'm actually really hoping that modules are the final catalyst for the community to realize that CMake is this generations' Autotools. ;) That said, I do think it's worthwhile that folks at least understand where Abyx is coming from. On Friday, February 26, 2016 at 11:57:44 AM UTC-8, Richard Smith wrote: > On Fri, Feb 26, 2016 at 7:23 AM, Abyx > > wrote: > 1) You manually list the dependencies between your libraries in your build rules. This is often a good thing for your code health, but may not be right for everyone. We're not talking about libraries! We're talking about translation units and modules. A single library could be made up of dozens and dozens (if not hundreds) of TUs and modules. > 2) Your build process scans for import statements. This is not really > any harder than scanning for #includes; no real parsing is required, > at most, you need tokenize and maybe preprocess (if you want 'import > MACRO_NAME' to work). The same is true for #includes. This is much harder than scanning for #include files. No good modern C++ build chain actually has a separate "scan for includes" step anymore. :) With headers, the dependency collection is just part of building. MSC, GCC, etc. all have flags that allow them to generate dependency chains while compiling, e.g. -MM -MF on GCC. The order of compilation of TUs is usually irrelevant; in fact, it's safe to build a dependent without even knowing the dependency exists, so long as all the dependee files already exist. The dependencies are used only for checking whether files are out of date relative to another file. In fact, A.cpp and B.cpp can be compiled in any order, or compiled in parallel, or compiled on two different machines, even if B.cpp has a dependency on A.h. With modules, this is no longer the case. The dependency graph must be present before any compilation begins, as building properly is now dependent on per-TU build artifacts (the .ifc files). Distributed builds require synchronization of the artifacts or island solving to ensure that files with dependency chains are all built on the same build node. This is because B.cpp no longer would depend on A.h, but instead would depend on A.ifc, which doesn't even exist until after A.cpp is compiled. Note in particular that this also means that with modules we've just serialized a previously parallel process, which is theoretically a build deoptimization - seemingly the exact opposite of what many of us are expecting or want out of modules! That said, your option 3 is the obvious solution. It's roughly what most other modern languages with module systems do, and C++ implementations and the build tools we use are going to have to modernize to adopt to a world where the compiler plays a more integral role in the build chain than it did before. > > 3) You ask your compiler to implicitly build (and cache) module > interfaces on demand. This requires that your compiler has some way to > map from an imported module name to the relevant interface file(s). > That could happen via some implementation-defined means (such as > Clang's module map files) or by making the module names directly > correspond to module interface files (as suggested in > http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0273r0.pdf). -- You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group. To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org. To post to this group, send email to std-proposals@isocpp.org. To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/d30abb01-3e49-424a-a410-556b60b1b833%40isocpp.org. ------=_Part_102_513709644.1456710066363 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I'll start off by saying that I don't think t= hat Abyx's worries are a significant problem. Yes, build systems will n= eed to change, but they've needed to change for 20 years now. If module= s are the impetus for the community to finally make building C++ not ridicu= lous, that's just fine with me. I'm actually really hoping that mod= ules are the final catalyst for the community to realize that CMake is this= generations' Autotools. ;)

That said, I do th= ink it's worthwhile that folks at least understand where Abyx is coming= from.

On Friday, February 26, 2016 at 11:57:4= 4 AM UTC-8, Richard Smith wrote:
On Fri, Feb 26, 2016 at 7:23 AM, Abyx <fl....@gmail.com> w= rote:

1) You manuall= y list the dependencies between your libraries in your=C2=A0
build rules. This is often a good thing for your= code health, but may=C2=A0
not be= right for everyone.=C2=A0=C2=A0

We're = not talking about libraries! We're talking about translation units and = modules.

A single library could be made up of doze= ns and dozens (if not hundreds) of TUs and modules.
=C2=A0
<= blockquote class=3D"gmail_quote" style=3D"margin: 0;margin-left: 0.8ex;bord= er-left: 1px #ccc solid;padding-left: 1ex;">2) Your build process scans for= import statements. This is not really
any harder than scanning for #includes; no real parsing is required,
at most, you need tokenize and maybe preprocess (if you want 'impor= t
MACRO_NAME' to work). The same is true for #includes. =C2=A0

This is much harder than scanning for #include fi= les. No good modern C++ build chain actually has a separate "scan for = includes" step anymore. :)

With headers, = the dependency collection is just part of building. MSC, GCC, etc. all have= flags that allow them to generate dependency chains while compiling, e.g. = -MM -MF <file> on GCC. The order of compilation of TUs is usually irr= elevant; in fact, it's safe to build a dependent without even knowing t= he dependency exists, so long as all the dependee files already exist. The = dependencies are used only for checking whether files are out of date relat= ive to another file.

In fact, A.cpp and B.cpp can = be compiled in any order, or compiled in parallel, or compiled on two diffe= rent machines, even if B.cpp has a dependency on A.h.

<= div>With modules, this is no longer the case. The dependency graph must be = present before any compilation begins, as building properly is now dependen= t on per-TU build artifacts (the .ifc files). Distributed builds require sy= nchronization of the artifacts or island solving to ensure that files with = dependency chains are all built on the same build node. This is because B.c= pp no longer would depend on A.h, but instead would depend on A.ifc, which = doesn't even exist until after A.cpp is compiled.

Note in particular that this also means that with modules we've= just serialized a previously parallel process, which is theoretically a bu= ild deoptimization - seemingly the exact opposite of what many of us are ex= pecting or want out of modules!

That said, your op= tion 3 is the obvious solution. It's roughly what most other modern lan= guages with module systems do, and C++ implementations and the build tools = we use are going to have to modernize to adopt to a world where the compile= r plays a more integral role in the build chain than it did before.
=C2=A0

3) You ask your compiler to implicitly build (and cache) module
interfaces on demand. This requires that your compiler has some way to
map from an imported module name to the relevant interface file(s).
That could happen via some implementation-defined means (such as
Clang's module map files) or by making the module names directly
correspond to module interface files (as suggested in
http://www.open-std= ..org/jtc1/sc22/wg21/docs/papers/2016/p0273r0.pdf).=C2=A0

--
You received this message because you are subscribed to the Google Groups &= quot;ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to std-proposa= ls+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.= com/a/isocpp.org/d/msgid/std-proposals/d30abb01-3e49-424a-a410-556b60b1b833= %40isocpp.org.
------=_Part_102_513709644.1456710066363-- ------=_Part_101_638399159.1456710066362-- .