220 24884 article Path: news.gmane.org!not-for-mail From: Richard Smith Newsgroups: gmane.comp.lang.c++.isocpp.proposals Subject: Re: Modules are hard for build tools Date: Wed, 2 Mar 2016 12:40:15 -0800 Lines: 199 Approved: news@gmane.org Message-ID: References: Reply-To: std-proposals@isocpp.org NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Trace: ger.gmane.org 1456951230 3395 80.91.229.3 (2 Mar 2016 20:40:30 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Wed, 2 Mar 2016 20:40:30 +0000 (UTC) To: "std-proposals@isocpp.org" Original-X-From: std-proposals+bncBDVNBJG4YAIBBME73W3AKGQELX6QF5Y@isocpp.org Wed Mar 02 21:40:21 2016 Return-path: Envelope-to: gclcip-std-proposals@m.gmane.org Original-Received: from mail-pf0-f199.google.com ([209.85.192.199]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1abDZD-0003Wh-BD for gclcip-std-proposals@m.gmane.org; Wed, 02 Mar 2016 21:40:19 +0100 Original-Received: by mail-pf0-f199.google.com with SMTP id 184sf1038159pff.3 for ; Wed, 02 Mar 2016 12:40:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:x-original-sender :x-original-authentication-results:reply-to:precedence:mailing-list :list-id:x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=4OD9mnhFME3FKKXhp4cT8oMPQj6jTvNcv8MK9qu4o/0=; b=B5GrIFbfDWIHnoqa1In93gkMTaeFYQcZ4WUVHHN0QCrFsssh/hHvQLbLnpurKq/8hi 8mvFIC4BpNwDVrbnDDqmeJq724ff5CTxW67nl+auOpZ2iAZDViSwxvbXPMgyvKbKU6bo q+v5zCrsf6oG1vJv1jVKmyVZQN5dloa0nvJuE481o+xZAvYjQTyCPXtONvkD+vzSJ1aX /wbGcuWJ5cKzBONFTr6lwwNCZvpxvd71yMb6wYiErtpP5rjuK7aMyO05SGx55M3DO7cP MPrBj5ABN3tMXQWXtsyXnASVGBlClRkURINeyyP/KXmwVH0gkd2HbMoySlgpastfAB7l 9yrQ== X-Gm-Message-State: AD7BkJIOgjY+HZOyQtBcOvNyRG+damj/s3FIMuatHgtsbz80TZ6zcFRBIeg/gr5owCjQlQ== X-Received: by 10.66.141.74 with SMTP id rm10mr23475147pab.16.1456951218397; Wed, 02 Mar 2016 12:40:18 -0800 (PST) X-BeenThere: std-proposals@isocpp.org Original-Received: by 10.140.81.145 with SMTP id f17ls3657113qgd.27.gmail; Wed, 02 Mar 2016 12:40:16 -0800 (PST) X-Received: by 10.31.33.199 with SMTP id h190mr21323297vkh.157.1456951216473; Wed, 02 Mar 2016 12:40:16 -0800 (PST) Original-Received: from mail-vk0-x229.google.com (mail-vk0-x229.google.com. [2607:f8b0:400c:c05::229]) by mx.google.com with ESMTPS id j12si23258571vka.29.2016.03.02.12.40.16 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 02 Mar 2016 12:40:16 -0800 (PST) Received-SPF: pass (google.com: domain of metafoo@gmail.com designates 2607:f8b0:400c:c05::229 as permitted sender) client-ip=2607:f8b0:400c:c05::229; Original-Received: by mail-vk0-x229.google.com with SMTP id c3so801586vkb.3 for ; Wed, 02 Mar 2016 12:40:16 -0800 (PST) X-Received: by 10.31.190.195 with SMTP id o186mr23170220vkf.100.1456951215851; Wed, 02 Mar 2016 12:40:15 -0800 (PST) Original-Sender: metafoo@gmail.com Original-Received: by 10.31.147.74 with HTTP; Wed, 2 Mar 2016 12:40:15 -0800 (PST) In-Reply-To: X-Original-Sender: richard@metafoo.co.uk X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of metafoo@gmail.com designates 2607:f8b0:400c:c05::229 as permitted sender) smtp.mailfrom=metafoo@gmail.com; dkim=pass header.i=@gmail.com Precedence: list Mailing-list: list std-proposals@isocpp.org; contact std-proposals+owners@isocpp.org List-ID: X-Spam-Checked-In-Group: std-proposals@isocpp.org X-Google-Group-Id: 399137483710 List-Post: , List-Help: , List-Archive: List-Subscribe: , List-Unsubscribe: , Xref: news.gmane.org gmane.comp.lang.c++.isocpp.proposals:24884 Archived-At: On Mon, Feb 29, 2016 at 7:30 AM, Abyx wrote: > Let's clarify how we're going to use modules. There is an opinion that > modules will completely replace header files. > Many code bases have header per class, e.g. widget.h, widget.cpp, button.h, > button.cpp, label.h, label.cpp > And because module exports can be only in a single file, that means we are > going to have (sub)module per class, e.g. modules widget, label, button, > etc. > > Not let's look at some decent mature code base, e.g. chromium - > https://code.google.com/p/chromium/codesearch#chromium/src/net/http/http_stream_factory_impl_job.cc > This file has around 50 fine-grained include directives. With modules it > will be 50 imports. > Some files have more than hundred includes, most of the files have more than > ten includes. > > No one in their right mind will copy all those dependencies into a build > script. > That rules out (1). That indicates you've made your modules far too small. Your widget, button, label, ... should probably form a single module; we don't need fine-grained includes and imports any more; that will make build performance *worse*, not better. If your set of UI widget classes instead forms a single module (perhaps with partitions for the individual classes, so you can still partition them into files as you see fit -- see p0273r0), it is not unreasonable for your build rules to say "I depend on that UI library". > 2) Build tool cannot scan for includes by itself. For imports it needs full > preprocessing (because of #ifdef and import statements in included files). That is the same problem that build tools see today with #include scanning, and yet it actually does work well in practice. > In order to do so it needs all the defines, include paths and implicit > (forced) includes from compiler-specific environment variables and global > (system) configuration files. > The ninja build tool has a nice explanation of this process - > https://ninja-build.org/manual.html#ref_headers That's describing a different process whereby the compiler generates dependencies as a side effect. That is not the include scanning process I was referring to. Please refer to the build tools I listed: http://scons.org/doc/1.1.0/HTML/scons-user/c3742.html http://www.boost.org/doc/libs/1_43_0/doc/html/jam/usage.html#jam.usage.operation.binding.headerscan https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/rules/cpp/IncludeScannable.java > And (3) would just embed a build tool into a compiler - we take a root .cpp > file which imports all the other modules, pass it to a compiler, and > compiler builds all the other TU. > That would be great except that now it's impossible to provide custom > compiler options to certain TU. Yes, that's true. It works for simple cases, but you'd want to pick a different strategy for more complex builds. This is the "I just want a simple Makefile that doesn't explicitly list any dependencies" approach. > On Monday, February 29, 2016 at 4:39:03 PM UTC+3, Richard Smith wrote: >> >> On Sun, Feb 28, 2016 at 5:41 PM, Sean Middleditch >> wrote: >> > I'll start off by saying that I don't think that Abyx's worries are a >> > significant problem. Yes, build systems will need to change, but they've >> > needed to change for 20 years now. If modules are the impetus for the >> > community to finally make building C++ not ridiculous, that's just fine >> > with >> > me. I'm actually really hoping that modules are the final catalyst for >> > the >> > community to realize that CMake is this generations' Autotools. ;) >> > >> > That said, I do think it's worthwhile that folks at least understand >> > where >> > Abyx is coming from. >> > >> > On Friday, February 26, 2016 at 11:57:44 AM UTC-8, Richard Smith wrote: >> >> >> >> On Fri, Feb 26, 2016 at 7:23 AM, Abyx wrote: >> > >> > >> >> 1) You manually list the dependencies between your libraries in your >> >> >> >> build rules. This is often a good thing for your code health, but may >> >> >> >> not be right for everyone. >> > >> > >> > We're not talking about libraries! We're talking about translation units >> > and >> > modules. >> >> We're talking about exactly the things that you configure in your >> build tool as a library -- that is, a small number of header files >> plus their corresponding source files. (These are not necessarily the >> same size as .so / .dll files / executables, which would typically be >> composed of a number of these build-system-level libraries.) >> >> > A single library could be made up of dozens and dozens (if not hundreds) >> > of >> > TUs and modules. >> > >> >> >> >> 2) Your build process scans for import statements. This is not really >> >> any harder than scanning for #includes; no real parsing is required, >> >> at most, you need tokenize and maybe preprocess (if you want 'import >> >> MACRO_NAME' to work). The same is true for #includes. >> > >> > >> > This is much harder than scanning for #include files. No good modern C++ >> > build chain actually has a separate "scan for includes" step anymore. :) >> >> The only way that's true is if you define "good modern" to make it >> true. Scons, Jam, Bazel, ... all support include scanning. >> >> > With headers, the dependency collection is just part of building. MSC, >> > GCC, >> > etc. all have flags that allow them to generate dependency chains while >> > compiling, e.g. -MM -MF on GCC. The order of compilation of TUs >> > is >> > usually irrelevant; in fact, it's safe to build a dependent without even >> > knowing the dependency exists, so long as all the dependee files already >> > exist. The dependencies are used only for checking whether files are out >> > of >> > date relative to another file. >> >> That approach does not work for large-scale distributed builds where >> the set of files that are inputs to the build must be known before the >> build action begins (so they can be staged to the build farm). >> >> > Note in particular that this also means that with modules we've just >> > serialized a previously parallel process, which is theoretically a build >> > deoptimization - seemingly the exact opposite of what many of us are >> > expecting or want out of modules! >> >> Yes, that's theoretically true, but does not match our experience in >> practice when building large existing codebases with (Clang's >> implementation of) modules. While use of modules does cause there to >> be a longer dependency chain for builds (compilation of Z.cpp depends >> on Y1.pcm, Y2.pcm, ... already having been compiled, which depend on >> X1.pcm, X2.pcm, ... already having been compiled, ...), and the total >> amount of work required is somewhat higher than compiling Z.cpp >> directly due to the serialization / deserialization overhead, those >> individual steps are parallelizable, the work to build them is reused, >> and despite the longer dependency chain we see significant build time >> reductions even for clean builds. >> >> > That said, your option 3 is the obvious solution. It's roughly what most >> > other modern languages with module systems do, and C++ implementations >> > and >> > the build tools we use are going to have to modernize to adopt to a >> > world >> > where the compiler plays a more integral role in the build chain than it >> > did >> > before. >> >> The downside is that option 3 parallelizes extremely poorly. If you >> want module builds to be parallelized, distributed, and each only >> performed once, you should look at a different option. If you only >> care about local builds, we've found that it can be effective. >> >> >> 3) You ask your compiler to implicitly build (and cache) module >> >> interfaces on demand. This requires that your compiler has some way to >> >> map from an imported module name to the relevant interface file(s). >> >> That could happen via some implementation-defined means (such as >> >> Clang's module map files) or by making the module names directly >> >> correspond to module interface files (as suggested in >> >> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0273r0.pdf). >> > >> > -- >> > You received this message because you are subscribed to the Google >> > Groups >> > "ISO C++ Standard - Future Proposals" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> > an >> > email to std-proposal...@isocpp.org. >> > To post to this group, send email to std-pr...@isocpp.org. >> > To view this discussion on the web visit >> > >> > https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/d30abb01-3e49-424a-a410-556b60b1b833%40isocpp.org. > > -- > You received this message because you are subscribed to the Google Groups > "ISO C++ Standard - Future Proposals" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to std-proposals+unsubscribe@isocpp.org. > To post to this group, send email to std-proposals@isocpp.org. > To view this discussion on the web visit > https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/d60dca15-396e-4886-92bf-3c9b79bbb829%40isocpp.org. -- You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group. To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org. To post to this group, send email to std-proposals@isocpp.org. To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAOfiQqm7AZxY2%2BwcQG8SCmFDPc7TYSGX3-FWH5cuifQ3bf9kWw%40mail.gmail.com. .