220 24830 article Path: news.gmane.org!not-for-mail From: Abyx Newsgroups: gmane.comp.lang.c++.isocpp.proposals Subject: Re: Modules are hard for build tools Date: Mon, 29 Feb 2016 07:32:16 -0800 (PST) Lines: 414 Approved: news@gmane.org Message-ID: References: Reply-To: std-proposals@isocpp.org NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_149_65210392.1456759936521" X-Trace: ger.gmane.org 1456759956 6180 80.91.229.3 (29 Feb 2016 15:32:36 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 29 Feb 2016 15:32:36 +0000 (UTC) To: ISO C++ Standard - Future Proposals Original-X-From: std-proposals+bncBCE4FQ4JYQLBBAOJ2G3AKGQEE7UXT5Y@isocpp.org Mon Feb 29 16:32:22 2016 Return-path: Envelope-to: gclcip-std-proposals@m.gmane.org Original-Received: from mail-oi0-f70.google.com ([209.85.218.70]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1aaPo2-0002xA-RX for gclcip-std-proposals@m.gmane.org; Mon, 29 Feb 2016 16:32:19 +0100 Original-Received: by mail-oi0-f70.google.com with SMTP id d205sf58536545oia.0 for ; Mon, 29 Feb 2016 07:32:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=isocpp-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:message-id:in-reply-to:references:subject:mime-version :x-original-sender:reply-to:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=j9UVfur1NEjEzxw+C/v5C8WQPUlbifupJ5pfB4oeBA0=; b=uQTNI/j77M4yRsj7ntryB9S0eMA9Gy+6RTT7URsNQ6YrmOtX08ocbFMiFKVNxf+6Fi YC4O03rRSY4R4CqVvxvYkLCGxUqOEA3IauLKvps/qwq69RKY7tjqqUvELxFbz2VOml3J sA+zGyWSPSg2PU/qeiB5LbXerQwSS4X4TqKh3DbfLKOHkrE3QrgHvPiiFxEkQhBqQh2y FIgUJLEnLxl7LJP4oQQ9Oa9hzsimrwu0S8yqXPq6i3tvK83czKa/9kPIbG53HdPL9ato nROs21Hgxizxo2LJw/v9Y12PBtycSmVPKz5bcnMZaO9gwDcnu3rcN7ONmgheDUoQOfFS PI4g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:message-id:in-reply-to:references:subject:mime-version :x-original-sender:reply-to:precedence:mailing-list:list-id :x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=j9UVfur1NEjEzxw+C/v5C8WQPUlbifupJ5pfB4oeBA0=; b=l7/8ckr2XKJv7vcEhvkebJ2QpLqNIcE+edx+NKQNfwIfIYBNHSPo5yDRTVzBpK3OMl f1NLOcHQLkxKKT69gnvgvNKzY14VM3lIEePCPIU4iZ4b9LAaRctf7d2WrC3YJYkor5Cs dgFkqOMNc3aR184ciKtOJDuh1wTFFqSvHbqH9M38qMBtCNLaTCGqMVx12hlxScraPu9B /xeQPgWJTaKfdA6470w6ONBXNuXoV4WL32IOS5Dw10oOK7uDP8Vn8n0vk2ZQxO5lOtFA 3x9yIWdkX2v5mS+SAARRiIFMJ2VNjOh/2AYhLgy6DNvjpG7zoVPQQAJpXhPxQXB9flS0 1GpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:message-id:in-reply-to:references :subject:mime-version:x-original-sender:reply-to:precedence :mailing-list:list-id:x-spam-checked-in-group:list-post:list-help :list-archive:list-subscribe:list-unsubscribe; bh=j9UVfur1NEjEzxw+C/v5C8WQPUlbifupJ5pfB4oeBA0=; b=hB7hmt37mg1k27rqOoWLOoMxikXeX2YX1db9deU2Zn7cXPSt9E03Ru33GynbNeUAgh Hj6rMdBtmZWgNW/fv7JAFTSqoIgwuKnNw/MWIiyCukTmGOLus6cTL49YW6LtCwhG00eb ISzZgfDYqmQKJHx8/ciI9bi1niqAEgqQzTlLI9bpBzfOOVj7oiznICaQ0aHnkDY9ZH57 Ysh4NyK9mHwU+9+E7bM0Q680r1AiWKBrlOvDL8GOxrYH27HlZ0BIUZTDxn35feGa1BQ6 Mq9e0SnjVh2TIHKxmute7PiDzVa+foAzl3HJnoIc75HB3CG5uD76O+YUO3F4ZCnNzCk3 oGYQ== X-Gm-Message-State: AG10YOTMpusHXpoZeSBYocI3FJ1eKbeNsXBoIjO51sRlaonagzzg1BSR736dxV3c1V+Acw== X-Received: by 10.107.11.23 with SMTP id v23mr18759289ioi.13.1456759937824; Mon, 29 Feb 2016 07:32:17 -0800 (PST) X-BeenThere: std-proposals@isocpp.org Original-Received: by 10.107.157.136 with SMTP id g130ls1339382ioe.56.gmail; Mon, 29 Feb 2016 07:32:17 -0800 (PST) X-Received: by 10.50.109.161 with SMTP id ht1mr178301igb.1.1456759937086; Mon, 29 Feb 2016 07:32:17 -0800 (PST) In-Reply-To: X-Original-Sender: fl.lllj@gmail.com Precedence: list Mailing-list: list std-proposals@isocpp.org; contact std-proposals+owners@isocpp.org List-ID: X-Spam-Checked-In-Group: std-proposals@isocpp.org X-Google-Group-Id: 399137483710 List-Post: , List-Help: , List-Archive: List-Subscribe: , List-Unsubscribe: , Xref: news.gmane.org gmane.comp.lang.c++.isocpp.proposals:24830 Archived-At: ------=_Part_149_65210392.1456759936521 Content-Type: multipart/alternative; boundary="----=_Part_150_135361035.1456759936521" ------=_Part_150_135361035.1456759936521 Content-Type: text/plain; charset=UTF-8 Let's clarify how we're going to use modules. There is an opinion that modules will completely replace header files. Many code bases have header per class, e.g. widget.h, widget.cpp, button.h, button.cpp, label.h, label.cpp And because module exports can be only in a single file, that means we are going to have (sub)module per class, e.g. modules widget, label, button, etc. Not let's look at some decent mature code base, e.g. chromium - https://code.google.com/p/chromium/codesearch#chromium/src/net/http/http_stream_factory_impl_job.cc This file has around 50 fine-grained include directives. With modules it will be 50 imports. Some files have more than hundred includes, most of the files have more than ten includes. No one in their right mind will copy all those dependencies into a build script. That rules out (1). 2) Build tool cannot scan for includes by itself. For imports it needs full preprocessing (because of #ifdef and import statements in included files). In order to do so it needs all the defines, include paths and implicit (forced) includes from compiler-specific environment variables and global (system) configuration files. The ninja build tool has a nice explanation of this process - https://ninja-build.org/manual.html#ref_headers There is one more issue - even if compiler would have a --list-imports switch, the build tool would have process files twice. First time - to get the dependencies, second time - to compile it after the dependencies were built. cc --list-imports a.cpp cc --list-imports dependency1.ixx ... cc --list-imports dependencyN.ixx cc dependencyN.ixx .... cc dependency1.ixx cc a.cpp And (3) would just embed a build tool into a compiler - we take a root .cpp file which imports all the other modules, pass it to a compiler, and compiler builds all the other TU. That would be great except that now it's impossible to provide custom compiler options to certain TU. Sorry for being so negative. On Monday, February 29, 2016 at 4:39:03 PM UTC+3, Richard Smith wrote: > > On Sun, Feb 28, 2016 at 5:41 PM, Sean Middleditch > > wrote: > > I'll start off by saying that I don't think that Abyx's worries are a > > significant problem. Yes, build systems will need to change, but they've > > needed to change for 20 years now. If modules are the impetus for the > > community to finally make building C++ not ridiculous, that's just fine > with > > me. I'm actually really hoping that modules are the final catalyst for > the > > community to realize that CMake is this generations' Autotools. ;) > > > > That said, I do think it's worthwhile that folks at least understand > where > > Abyx is coming from. > > > > On Friday, February 26, 2016 at 11:57:44 AM UTC-8, Richard Smith wrote: > >> > >> On Fri, Feb 26, 2016 at 7:23 AM, Abyx wrote: > > > > > >> 1) You manually list the dependencies between your libraries in your > >> > >> build rules. This is often a good thing for your code health, but may > >> > >> not be right for everyone. > > > > > > We're not talking about libraries! We're talking about translation units > and > > modules. > > We're talking about exactly the things that you configure in your > build tool as a library -- that is, a small number of header files > plus their corresponding source files. (These are not necessarily the > same size as .so / .dll files / executables, which would typically be > composed of a number of these build-system-level libraries.) > > > A single library could be made up of dozens and dozens (if not hundreds) > of > > TUs and modules. > > > >> > >> 2) Your build process scans for import statements. This is not really > >> any harder than scanning for #includes; no real parsing is required, > >> at most, you need tokenize and maybe preprocess (if you want 'import > >> MACRO_NAME' to work). The same is true for #includes. > > > > > > This is much harder than scanning for #include files. No good modern C++ > > build chain actually has a separate "scan for includes" step anymore. :) > > The only way that's true is if you define "good modern" to make it > true. Scons, Jam, Bazel, ... all support include scanning. > > > With headers, the dependency collection is just part of building. MSC, > GCC, > > etc. all have flags that allow them to generate dependency chains while > > compiling, e.g. -MM -MF on GCC. The order of compilation of TUs > is > > usually irrelevant; in fact, it's safe to build a dependent without even > > knowing the dependency exists, so long as all the dependee files already > > exist. The dependencies are used only for checking whether files are out > of > > date relative to another file. > > That approach does not work for large-scale distributed builds where > the set of files that are inputs to the build must be known before the > build action begins (so they can be staged to the build farm). > > > Note in particular that this also means that with modules we've just > > serialized a previously parallel process, which is theoretically a build > > deoptimization - seemingly the exact opposite of what many of us are > > expecting or want out of modules! > > Yes, that's theoretically true, but does not match our experience in > practice when building large existing codebases with (Clang's > implementation of) modules. While use of modules does cause there to > be a longer dependency chain for builds (compilation of Z.cpp depends > on Y1.pcm, Y2.pcm, ... already having been compiled, which depend on > X1.pcm, X2.pcm, ... already having been compiled, ...), and the total > amount of work required is somewhat higher than compiling Z.cpp > directly due to the serialization / deserialization overhead, those > individual steps are parallelizable, the work to build them is reused, > and despite the longer dependency chain we see significant build time > reductions even for clean builds. > > > That said, your option 3 is the obvious solution. It's roughly what most > > other modern languages with module systems do, and C++ implementations > and > > the build tools we use are going to have to modernize to adopt to a > world > > where the compiler plays a more integral role in the build chain than it > did > > before. > > The downside is that option 3 parallelizes extremely poorly. If you > want module builds to be parallelized, distributed, and each only > performed once, you should look at a different option. If you only > care about local builds, we've found that it can be effective. > > >> 3) You ask your compiler to implicitly build (and cache) module > >> interfaces on demand. This requires that your compiler has some way to > >> map from an imported module name to the relevant interface file(s). > >> That could happen via some implementation-defined means (such as > >> Clang's module map files) or by making the module names directly > >> correspond to module interface files (as suggested in > >> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0273r0.pdf). > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "ISO C++ Standard - Future Proposals" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to std-proposal...@isocpp.org . > > To post to this group, send email to std-pr...@isocpp.org . > > > To view this discussion on the web visit > > > https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/d30abb01-3e49-424a-a410-556b60b1b833%40isocpp.org. > > -- You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group. To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org. To post to this group, send email to std-proposals@isocpp.org. To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/bc32cd95-7011-479c-85eb-a4ebe1777cd8%40isocpp.org. ------=_Part_150_135361035.1456759936521 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Let's clarify how we're going to use modules. Ther= e is an opinion that modules will completely replace header files.
Many = code bases have header per class, e.g. widget.h, widget.cpp, button.h, butt= on.cpp, label.h, label.cpp
And because module exports can be only in a s= ingle file, that means we are going to have (sub)module per class, e.g. mod= ules widget, label, button, etc.

Not let's look at some decent m= ature code base, e.g. chromium - https://code.google.com/p/chromium/codesea= rch#chromium/src/net/http/http_stream_factory_impl_job.cc
This file has = around 50 fine-grained include directives. With modules it will be 50 impor= ts.
Some files have more than hundred includes, most of the files have m= ore than ten includes.

No one in their right mind will copy all thos= e dependencies into a build script.
That rules out (1).

2) Build = tool cannot scan for includes by itself. For imports it needs full preproce= ssing (because of #ifdef and import statements in included files).
In or= der to do so it needs all the defines, include paths and implicit (forced) = includes from compiler-specific environment variables and global (system) c= onfiguration files.
The ninja build tool has a nice explanation of this = process - https://ninja-build.org/manual.html#ref_headers

There is o= ne more issue - even if compiler would have a --list-imports switch, the bu= ild tool would have process files twice.
First time - to get the depende= ncies, second time - to compile it after the dependencies were built.
cc --list-imports a.cpp
cc --list-imports dependency1.ixx
..
cc = --list-imports dependencyN.ixx
cc dependencyN.ixx
...
cc dependenc= y1.ixx
cc a.cpp

And (3) would just embed a build tool into a comp= iler - we take a root .cpp file which imports all the other modules, pass i= t to a compiler, and compiler builds all the other TU.
That would be gre= at except that now it's impossible to provide custom compiler options t= o certain TU.

Sorry for being so negative.


On Monday, Feb= ruary 29, 2016 at 4:39:03 PM UTC+3, Richard Smith wrote:
On Sun, Feb 28, 2016 at 5:41 PM, Sean Middledit= ch
<sean.mid...@gmail.com> wrote:
> I'll start off by saying that I don't think that Abyx'= s worries are a
> significant problem. Yes, build systems will need to change, but t= hey've
> needed to change for 20 years now. If modules are the impetus for = the
> community to finally make building C++ not ridiculous, that's = just fine with
> me. I'm actually really hoping that modules are the final cata= lyst for the
> community to realize that CMake is this generations' Autotools= .. ;)
>
> That said, I do think it's worthwhile that folks at least unde= rstand where
> Abyx is coming from.
>
> On Friday, February 26, 2016 at 11:57:44 AM UTC-8, Richard Smith w= rote:
>>
>> On Fri, Feb 26, 2016 at 7:23 AM, Abyx <fl....@gmail.com<= /a>> wrote:
>
>
>> 1) You manually list the dependencies between your libraries i= n your
>>
>> build rules. This is often a good thing for your code health, = but may
>>
>> not be right for everyone.
>
>
> We're not talking about libraries! We're talking about tra= nslation units and
> modules.

We're talking about exactly the things that you configure in your
build tool as a library -- that is, a small number of header files
plus their corresponding source files. (These are not necessarily the
same size as .so / .dll files / executables, which would typically be
composed of a number of these build-system-level libraries.)

> A single library could be made up of dozens and dozens (if not hun= dreds) of
> TUs and modules.
>
>>
>> 2) Your build process scans for import statements. This is not= really
>> any harder than scanning for #includes; no real parsing is req= uired,
>> at most, you need tokenize and maybe preprocess (if you want &= #39;import
>> MACRO_NAME' to work). The same is true for #includes.
>
>
> This is much harder than scanning for #include files. No good mode= rn C++
> build chain actually has a separate "scan for includes" = step anymore. :)

The only way that's true is if you define "good modern" t= o make it
true. Scons, Jam, Bazel, ... all support include scanning.

> With headers, the dependency collection is just part of building. = MSC, GCC,
> etc. all have flags that allow them to generate dependency chains = while
> compiling, e.g. -MM -MF <file> on GCC. The order of compilat= ion of TUs is
> usually irrelevant; in fact, it's safe to build a dependent wi= thout even
> knowing the dependency exists, so long as all the dependee files a= lready
> exist. The dependencies are used only for checking whether files a= re out of
> date relative to another file.

That approach does not work for large-scale distributed builds where
the set of files that are inputs to the build must be known before the
build action begins (so they can be staged to the build farm).

> Note in particular that this also means that with modules we'v= e just
> serialized a previously parallel process, which is theoretically a= build
> deoptimization - seemingly the exact opposite of what many of us a= re
> expecting or want out of modules!

Yes, that's theoretically true, but does not match our experience i= n
practice when building large existing codebases with (Clang's
implementation of) modules. While use of modules does cause there to
be a longer dependency chain for builds (compilation of Z.cpp depends
on Y1.pcm, Y2.pcm, ... already having been compiled, which depend on
X1.pcm, X2.pcm, ... already having been compiled, ...), and the total
amount of work required is somewhat higher than compiling Z.cpp
directly due to the serialization / deserialization overhead, those
individual steps are parallelizable, the work to build them is reused,
and despite the longer dependency chain we see significant build time
reductions even for clean builds.

> That said, your option 3 is the obvious solution. It's roughly= what most
> other modern languages with module systems do, and C++ implementat= ions and
> the build tools we use are going to have to modernize to adopt to = a world
> where the compiler plays a more integral role in the build chain t= han it did
> before.

The downside is that option 3 parallelizes extremely poorly. If you
want module builds to be parallelized, distributed, and each only
performed once, you should look at a different option. If you only
care about local builds, we've found that it can be effective.

>> 3) You ask your compiler to implicitly build (and cache) modul= e
>> interfaces on demand. This requires that your compiler has som= e way to
>> map from an imported module name to the relevant interface fil= e(s).
>> That could happen via some implementation-defined means (such = as
>> Clang's module map files) or by making the module names di= rectly
>> correspond to module interface files (as suggested in
>> http://www= ..open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0273r0.pdf).
>
> --
> You received this message because you are subscribed to the Google= Groups
> "ISO C++ Standard - Future Proposals" group.
> To unsubscribe from this group and stop receiving emails from it, = send an
> email to std-proposal...@isocpp.org.
> To post to this group, send email to std-pr...@isocpp.org.
> To view this discussion on the web visit
> https://groups.google.com/a/isocp= p.org/d/msgid/std-proposals/d30abb01-3e49-424a-a410-556b60b1b833%= 40isocpp.org.

--
You received this message because you are subscribed to the Google Groups &= quot;ISO C++ Standard - Future Proposals" group.
To unsubscribe from this group and stop receiving emails from it, send an e= mail to std-proposa= ls+unsubscribe@isocpp.org.
To post to this group, send email to std-proposals@isocpp.org.
To view this discussion on the web visit https://groups.google.= com/a/isocpp.org/d/msgid/std-proposals/bc32cd95-7011-479c-85eb-a4ebe1777cd8= %40isocpp.org.
------=_Part_150_135361035.1456759936521-- ------=_Part_149_65210392.1456759936521-- .