220 24827 article Path: news.gmane.org!not-for-mail From: Richard Smith Newsgroups: gmane.comp.lang.c++.isocpp.proposals Subject: Re: Modules are hard for build tools Date: Mon, 29 Feb 2016 05:39:00 -0800 Lines: 112 Approved: news@gmane.org Message-ID: References: Reply-To: std-proposals@isocpp.org NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 X-Trace: ger.gmane.org 1456753150 18333 80.91.229.3 (29 Feb 2016 13:39:10 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Mon, 29 Feb 2016 13:39:10 +0000 (UTC) To: "std-proposals@isocpp.org" Original-X-From: std-proposals+bncBDVNBJG4YAIBB5ET2G3AKGQEC3DH4UA@isocpp.org Mon Feb 29 14:39:05 2016 Return-path: Envelope-to: gclcip-std-proposals@m.gmane.org Original-Received: from mail-ig0-f200.google.com ([209.85.213.200]) by plane.gmane.org with esmtp (Exim 4.69) (envelope-from ) id 1aaO2S-0000kP-3U for gclcip-std-proposals@m.gmane.org; Mon, 29 Feb 2016 14:39:04 +0100 Original-Received: by mail-ig0-f200.google.com with SMTP id rx16sf230157744igc.3 for ; Mon, 29 Feb 2016 05:39:03 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:x-original-sender :x-original-authentication-results:reply-to:precedence:mailing-list :list-id:x-spam-checked-in-group:list-post:list-help:list-archive :list-subscribe:list-unsubscribe; bh=E7UVtSZ1TqaVSTkqEt8QWHDYwiGHY4ed4RfvfezMVIA=; b=XrUAVfHduJzDUCpWk/3cE+G/w/G7v5LijCsqkh0A0gHtzPuchputptoe4ebOF7Cjls /QztD0KShKKTCkw/xhMgsqIdMTUOJtyAqGpjdRGE2xcOcCRgMiQxxJ6gxcCUibktxzD4 2+id7u09NHTWjvo8H6n5v6d41VR628hAuDtpnczVHc4XkpziNeLX/Z++yWRGVyegwT2f BRIYIoAnNidczD5VVQW7LGAPW7YFcwmiNlVDLkfFnW3f9ZqYayHJ9oebTSkpFTzJbDF1 6V8JF5RIBZq9l26kqX/PwGROxqCQDdiaH+QVtM25++nyUgGAL9rQabwZMMQ79T5VNb4c tBVA== X-Gm-Message-State: AG10YORlyW/jNT7t112dtD7Y9L/tvY3rXcwJoRqWXeB+sC42BZQ+fqmZtQWzjau+3N3byA== X-Received: by 10.107.138.211 with SMTP id c80mr18253354ioj.20.1456753143129; Mon, 29 Feb 2016 05:39:03 -0800 (PST) X-BeenThere: std-proposals@isocpp.org Original-Received: by 10.140.30.98 with SMTP id c89ls546180qgc.26.gmail; Mon, 29 Feb 2016 05:39:00 -0800 (PST) X-Received: by 10.31.9.72 with SMTP id 69mr9171099vkj.126.1456753140465; Mon, 29 Feb 2016 05:39:00 -0800 (PST) Original-Received: from mail-vk0-x234.google.com (mail-vk0-x234.google.com. [2607:f8b0:400c:c05::234]) by mx.google.com with ESMTPS id e21si15794217vkd.21.2016.02.29.05.39.00 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Feb 2016 05:39:00 -0800 (PST) Received-SPF: pass (google.com: domain of metafoo@gmail.com designates 2607:f8b0:400c:c05::234 as permitted sender) client-ip=2607:f8b0:400c:c05::234; Original-Received: by mail-vk0-x234.google.com with SMTP id e6so133370556vkh.2 for ; Mon, 29 Feb 2016 05:39:00 -0800 (PST) X-Received: by 10.31.190.195 with SMTP id o186mr12154182vkf.100.1456753140083; Mon, 29 Feb 2016 05:39:00 -0800 (PST) Original-Sender: metafoo@gmail.com Original-Received: by 10.31.147.74 with HTTP; Mon, 29 Feb 2016 05:39:00 -0800 (PST) In-Reply-To: X-Original-Sender: richard@metafoo.co.uk X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of metafoo@gmail.com designates 2607:f8b0:400c:c05::234 as permitted sender) smtp.mailfrom=metafoo@gmail.com; dkim=pass header.i=@gmail.com Precedence: list Mailing-list: list std-proposals@isocpp.org; contact std-proposals+owners@isocpp.org List-ID: X-Spam-Checked-In-Group: std-proposals@isocpp.org X-Google-Group-Id: 399137483710 List-Post: , List-Help: , List-Archive: List-Subscribe: , List-Unsubscribe: , Xref: news.gmane.org gmane.comp.lang.c++.isocpp.proposals:24827 Archived-At: On Sun, Feb 28, 2016 at 5:41 PM, Sean Middleditch wrote: > I'll start off by saying that I don't think that Abyx's worries are a > significant problem. Yes, build systems will need to change, but they've > needed to change for 20 years now. If modules are the impetus for the > community to finally make building C++ not ridiculous, that's just fine with > me. I'm actually really hoping that modules are the final catalyst for the > community to realize that CMake is this generations' Autotools. ;) > > That said, I do think it's worthwhile that folks at least understand where > Abyx is coming from. > > On Friday, February 26, 2016 at 11:57:44 AM UTC-8, Richard Smith wrote: >> >> On Fri, Feb 26, 2016 at 7:23 AM, Abyx wrote: > > >> 1) You manually list the dependencies between your libraries in your >> >> build rules. This is often a good thing for your code health, but may >> >> not be right for everyone. > > > We're not talking about libraries! We're talking about translation units and > modules. We're talking about exactly the things that you configure in your build tool as a library -- that is, a small number of header files plus their corresponding source files. (These are not necessarily the same size as .so / .dll files / executables, which would typically be composed of a number of these build-system-level libraries.) > A single library could be made up of dozens and dozens (if not hundreds) of > TUs and modules. > >> >> 2) Your build process scans for import statements. This is not really >> any harder than scanning for #includes; no real parsing is required, >> at most, you need tokenize and maybe preprocess (if you want 'import >> MACRO_NAME' to work). The same is true for #includes. > > > This is much harder than scanning for #include files. No good modern C++ > build chain actually has a separate "scan for includes" step anymore. :) The only way that's true is if you define "good modern" to make it true. Scons, Jam, Bazel, ... all support include scanning. > With headers, the dependency collection is just part of building. MSC, GCC, > etc. all have flags that allow them to generate dependency chains while > compiling, e.g. -MM -MF on GCC. The order of compilation of TUs is > usually irrelevant; in fact, it's safe to build a dependent without even > knowing the dependency exists, so long as all the dependee files already > exist. The dependencies are used only for checking whether files are out of > date relative to another file. That approach does not work for large-scale distributed builds where the set of files that are inputs to the build must be known before the build action begins (so they can be staged to the build farm). > Note in particular that this also means that with modules we've just > serialized a previously parallel process, which is theoretically a build > deoptimization - seemingly the exact opposite of what many of us are > expecting or want out of modules! Yes, that's theoretically true, but does not match our experience in practice when building large existing codebases with (Clang's implementation of) modules. While use of modules does cause there to be a longer dependency chain for builds (compilation of Z.cpp depends on Y1.pcm, Y2.pcm, ... already having been compiled, which depend on X1.pcm, X2.pcm, ... already having been compiled, ...), and the total amount of work required is somewhat higher than compiling Z.cpp directly due to the serialization / deserialization overhead, those individual steps are parallelizable, the work to build them is reused, and despite the longer dependency chain we see significant build time reductions even for clean builds. > That said, your option 3 is the obvious solution. It's roughly what most > other modern languages with module systems do, and C++ implementations and > the build tools we use are going to have to modernize to adopt to a world > where the compiler plays a more integral role in the build chain than it did > before. The downside is that option 3 parallelizes extremely poorly. If you want module builds to be parallelized, distributed, and each only performed once, you should look at a different option. If you only care about local builds, we've found that it can be effective. >> 3) You ask your compiler to implicitly build (and cache) module >> interfaces on demand. This requires that your compiler has some way to >> map from an imported module name to the relevant interface file(s). >> That could happen via some implementation-defined means (such as >> Clang's module map files) or by making the module names directly >> correspond to module interface files (as suggested in >> http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0273r0.pdf). > > -- > You received this message because you are subscribed to the Google Groups > "ISO C++ Standard - Future Proposals" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to std-proposals+unsubscribe@isocpp.org. > To post to this group, send email to std-proposals@isocpp.org. > To view this discussion on the web visit > https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/d30abb01-3e49-424a-a410-556b60b1b833%40isocpp.org. -- You received this message because you are subscribed to the Google Groups "ISO C++ Standard - Future Proposals" group. To unsubscribe from this group and stop receiving emails from it, send an email to std-proposals+unsubscribe@isocpp.org. To post to this group, send email to std-proposals@isocpp.org. To view this discussion on the web visit https://groups.google.com/a/isocpp.org/d/msgid/std-proposals/CAOfiQqnL7tw3y6v5D-hBfbxQhE-w8EXDC-Nr%2BYjXowVg97DJnA%40mail.gmail.com. .