Standard Library Modules Bug Bash

⚠️ Outdated!

This page is outdated now that the Standard Library Modules have been merged into microsoft/STL main.

Results

Scores

The bug bash is finished! Thanks to everyone who tried out the branch and reported issues. Here are the final scores:

Contributor	Points
@StephanTLavavej	37 points
@nico-engels	15 points
@JMazurkiewicz	14 points
@cpplearner	7 points
@xiaosa-zhz	5 points

The Saga Continues...

I'll still accept bug reports, so I've preserved the instructions below. Updates:

• Please use the import-std branch, not the bug-bash branch.
• The import-std branch provides no guarantees of stability - as I work, I'll be rebasing it and changing commits.
• The arbitrary internet points are frozen - bug reports are now for the good of the C++ ecosystem. 😸

Overview

About

I'm working on the Standard Library Modules feature. It's really exciting, as it'll make code faster to compile and simpler to write - you'll be able to import std; instead of including dozens of headers. I've already gone through over 145,000 lines of the STL's headers and manually added over 3,700 exports. One of the major challenges is that C++ is a complex language, the STL uses it in complex ways, and digesting that usage into a new serialized format has revealed many corner cases in the compiler. I've already reported over 50 compiler bugs while working on this (and the earlier Standard Library Header Units, which uses much of the same compiler machinery). Almost all of those compiler bugs have been fixed thanks to the compiler front-end team's tireless work, so compiler stability has greatly improved, but more work remains to be done.

That's where this bug bash comes in. I want Standard Library Modules to ship with a very high level of quality, so users will have a great experience and can begin modularizing their codebases. I've already tested some machinery from every header, but I'm sure there are more bugs lurking that will be revealed by more usage. So, I'm inviting you to try out a branch from my personal fork, and report any issues you encounter that are specific to Standard Library Modules versus classic includes. I'll take care of the rest (analyzing them to see whether the compiler or the library needs to be fixed, and whether a library workaround is possible).

Links

• P2465R3 Standard Library Modules std and std.compat
  • This is the latest version of the proposal, which was accepted on 2022-07-25.
• N4910 C++ Working Draft
  • This is the latest version of the C++23 Working Draft, released 2022-03-17.
• MSVC Compiler Options
  • This explains /MTd, /Od, etc.
• microsoft/STL#1694 Standard Library Header Units and Modules - tracking issue
  • This records the many compiler bugs that I've encountered and reported while working on this feature.

Getting Started

Install VS Preview

1. Install the latest VS Community 2022 Preview.
   • You must use the latest Preview (currently 17.4 Preview 2), which contains important compiler bugfixes. The latest production version (currently 17.3) will not work for this bug bash.
   • The edition doesn't matter, if you happen to already have Professional or Enterprise installed.
2. The VS Installer will display the Workloads tab. Scroll down and select "Desktop development with C++".
3. In the "Installation details" pane to the right, deselect the Windows 10 SDK (10.0.19041.0).
   • The STL's build will reject this version.
4. Scroll down and select the Windows 11 SDK (10.0.22000.0).
   • The STL's build requires this version, but your OS doesn't need to be Windows 11; Windows 10 should work fine.
   • We haven't tested the newer version of the Windows 11 SDK (10.0.22621.0) when building the STL, although in theory it should work.
5. Begin the installation.
   • The installer should eventually say "Visual Studio has been successfully installed. We recommend rebooting soon to clean up any remaining files."
6. Reboot.

Clone the STL

These are quick-start instructions for the bug bash, not meant for normal STL development.

1. Install git, if you haven't already.
   • If you're unfamiliar with the options, you can accept the defaults, although I have two recommendations:
     • Select a text editor you're comfortable with.
     • Select "Checkout as-is, commit as-is" for line endings.
2. Open an "x64 Native Tools Command Prompt for VS 2022 Preview".
3. Change to a working directory where you'd like to clone the STL. In this example, I'll use C:\GitHub.

4. git clone https://github.com/StephanTLavavej/STL --branch import-std

5. cd STL

6. git submodule update --init --progress --depth=1 boost-math

Build the STL

1. cmake -G Ninja -S . -B out\x64 -DBUILD_TESTING=OFF

2. ninja -C out\x64

3. out\x64\set_environment.bat

   • After a successful build, set_environment.bat adds the freshly-built STL to the INCLUDE and LIB environment variables so it can be used.

Build and Test the Modules

1. cl /EHsc /nologo /W4 /std:c++latest /MTd /Od /c stl\modules\std.ixx stl\modules\std.compat.ixx

2. cl /EHsc /nologo /W4 /std:c++latest /MTd /Od meow.cpp std.obj

3. meow.exe

Congratulations - you're now one of the first users of import std;! 🎉 😸

Expected Output

Click to expand expected output:

C:\GitHub>git clone https://github.com/StephanTLavavej/STL --branch import-std
Cloning into 'STL'...
remote: Enumerating objects: 21015, done.
remote: Counting objects: 100% (564/564), done.
remote: Compressing objects: 100% (217/217), done.
remote: Total 21015 (delta 366), reused 501 (delta 344), pack-reused 20451
Receiving objects: 100% (21015/21015), 15.54 MiB | 18.72 MiB/s, done.
Resolving deltas: 100% (14487/14487), done.
Updating files: 100% (2143/2143), done.

C:\GitHub>cd STL

C:\GitHub\STL>git submodule update --init --progress --depth=1 boost-math
Submodule 'boost-math' (https://github.com/boostorg/math.git) registered for path 'boost-math'
Cloning into 'C:/GitHub/STL/boost-math'...
remote: Enumerating objects: 4450, done.
remote: Counting objects: 100% (4450/4450), done.
remote: Compressing objects: 100% (3213/3213), done.
remote: Total 4450 (delta 2370), reused 2331 (delta 1204), pack-reused 0
Receiving objects: 100% (4450/4450), 25.33 MiB | 25.53 MiB/s, done.
Resolving deltas: 100% (2370/2370), done.
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
remote: Enumerating objects: 2118, done.
remote: Counting objects: 100% (2118/2118), done.
remote: Compressing objects: 100% (977/977), done.
remote: Total 1120 (delta 993), reused 206 (delta 133), pack-reused 0
Receiving objects: 100% (1120/1120), 2.77 MiB | 8.66 MiB/s, done.
Resolving deltas: 100% (993/993), completed with 846 local objects.
From https://github.com/boostorg/math
 * branch            88c4b8d661dbe15be7317f04f1eccae0c9c39980 -> FETCH_HEAD
Submodule path 'boost-math': checked out '88c4b8d661dbe15be7317f04f1eccae0c9c39980'

C:\GitHub\STL>cmake -G Ninja -S . -B out\x64 -DBUILD_TESTING=OFF
-- The CXX compiler identification is MSVC 19.33.31627.1
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Preview/VC/Tools/MSVC/14.33.31627/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test WINDOWS_SDK_VERSION_CHECK
-- Performing Test WINDOWS_SDK_VERSION_CHECK - Success
-- The ASM_MASM compiler identification is MSVC
-- Found assembler: C:/Program Files/Microsoft Visual Studio/2022/Preview/VC/Tools/MSVC/14.33.31627/bin/Hostx64/x64/ml64.exe
-- Searching for VS clang-format
-- Searching for VS clang-format - found
-- Boost.Math: standalone mode ON
-- Configuring done
-- Generating done
-- Build files have been written to: C:/GitHub/STL/out/x64

C:\GitHub\STL>ninja -C out\x64
ninja: Entering directory `out\x64'
[1/1014] Building ASM_MASM object stl\CMakeFiles\stl_alias_objects.dir\src\alias_init_once_begin_initialize.asm.obj
Microsoft (R) Macro Assembler (x64) Version 14.33.31627.1
Copyright (C) Microsoft Corporation.  All rights reserved.

 Assembling: C:\GitHub\STL\stl\src\alias_init_once_begin_initialize.asm
[2/1014] Building ASM_MASM object stl\CMakeFiles\stl_alias_objects.dir\src\alias_init_once_complete.asm.obj
Microsoft (R) Macro Assembler (x64) Version 14.33.31627.1
Copyright (C) Microsoft Corporation.  All rights reserved.

 Assembling: C:\GitHub\STL\stl\src\alias_init_once_complete.asm
[1014/1014] Linking CXX static library out\lib\amd64\libcpmtd0.lib

C:\GitHub\STL>out\x64\set_environment.bat

C:\GitHub\STL>cl /EHsc /nologo /W4 /std:c++latest /MTd /Od /c stl\modules\std.ixx stl\modules\std.compat.ixx
std.ixx
std.compat.ixx
Generating Code...

C:\GitHub\STL>cl /EHsc /nologo /W4 /std:c++latest /MTd /Od meow.cpp std.obj
meow.cpp

C:\GitHub\STL>meow.exe
Hello, modules world!
11 22 33
Testing <algorithm>.
Testing <any>.
Testing <array>.
Testing <atomic>.
Testing <barrier>.
Testing <bit>.
Testing <bitset>.
Testing <charconv>.
Testing <chrono>.
Testing <codecvt>.
Testing <compare>.
Testing <complex>.
Testing <concepts>.
Testing <condition_variable>.
Testing <coroutine>.
Testing <deque>.
Testing <exception>.
Testing <execution>.
Testing <expected>.
Testing <filesystem>.
Testing <format>.
Testing <forward_list>.
Testing <fstream>.
Testing <functional>.
Testing <future>.
Testing <initializer_list>.
Testing <iomanip>.
Testing <ios>.
Testing <iosfwd>.
Testing <iostream>.
Cute fluffy kittens!
Testing <istream>.
Testing <iterator>.
Testing <latch>.
Testing <limits>.
Testing <list>.
Testing <locale>.
Testing <map>.
Testing <memory_resource>.
Testing <memory>.
Testing <mutex>.
Testing <new>.
Testing <numbers>.
Testing <numeric>.
Testing <optional>.
Testing <ostream>.
Testing <queue>.
Testing <random>.
Testing <ranges>.
Testing <ratio>.
Testing <regex>.
Testing <scoped_allocator>.
Testing <semaphore>.
Testing <set>.
Testing <shared_mutex>.
Testing <source_location>.
Testing <span>.
Testing <spanstream>.
Testing <sstream>.
Testing <stack>.
Testing <stacktrace>.
meow+0x6184
Testing <stdexcept>.
Testing <stop_token>.
Testing <streambuf>.
Testing <string_view>.
Testing <string>.
Testing <strstream>.
Testing <syncstream>.
Kittens are cute and fluffy!
Testing <system_error>.
Testing <thread>.
Testing <tuple>.
Testing <type_traits>.
Testing <typeindex>.
Testing <typeinfo>.
Testing <unordered_map>.
Testing <unordered_set>.
Testing <utility>.
Testing <valarray>.
Testing <variant>.
Testing <vector>.
Skipping <version> (nothing to do, macros unavailable).
Done!

Building in a Different Directory

When consuming the std named module, the compiler needs std.ifc and the linker needs std.obj. In the example above, I explicitly mentioned std.obj, but I didn't explicitly mention std.ifc. It worked because std.ifc was built in the local directory.

If you're building in a different directory (e.g. a larger project), you'll need to tell the compiler where to find std.ifc. The compiler option for that is /reference. Both /reference LOCATION_OF\std.ifc and /reference std=LOCATION_OF\std.ifc will work; as the documentation explains, the form that mentions std= will be slightly faster as it tells the compiler what the module name is (otherwise, the compiler has to open the IFC file to discover that).

Here's an example where I created a subdirectory ELSEWHERE, and compiled using the files in the parent directory:

C:\GitHub\STL\ELSEWHERE>cl /EHsc /nologo /W4 /std:c++latest /reference ..\std.ifc /MTd /Od ..\meow.cpp ..\std.obj
meow.cpp

C:\GitHub\STL\ELSEWHERE>cl /EHsc /nologo /W4 /std:c++latest /reference std=..\std.ifc /MTd /Od ..\meow.cpp ..\std.obj
meow.cpp

C:\GitHub\STL\ELSEWHERE>

Classic Includes vs. Named Modules

Success

Here's an example where they both work:

C:\GitHub\STL>type purr.cpp

#include <assert.h>

#ifdef USE_CLASSIC_INCLUDES
    #include <cstdio>
    #include <vector>
#else
    import std;
#endif

int main() {
#ifdef USE_CLASSIC_INCLUDES
    std::puts("Classic includes.");
#else
    std::puts("Named modules.");
#endif

    std::vector<int> vec{11, 22, 33};
    assert(vec[1] == 22);

    std::printf("vec: ");
    for (const auto& elem : vec) {
        std::printf("%d ", elem);
    }
    std::printf("\n");
}

C:\GitHub\STL>cl /EHsc /nologo /W4 /std:c++latest /MTd /Od purr.cpp /DUSE_CLASSIC_INCLUDES && purr.exe
purr.cpp
Classic includes.
vec: 11 22 33

C:\GitHub\STL>cl /EHsc /nologo /W4 /std:c++latest /MTd /Od purr.cpp std.obj && purr.exe
purr.cpp
Named modules.
vec: 11 22 33

Failure

Here's an example where classic includes work, but named modules emit a compiler error:

C:\GitHub\STL>type woof.cpp

#include <assert.h>

#ifdef USE_CLASSIC_INCLUDES
    #include <cerrno>
    #include <cmath>
    #include <cstdio>
#else
    import std;
#endif

int main() {
#ifdef USE_CLASSIC_INCLUDES
    std::puts("Classic includes.");
#else
    std::puts("Named modules.");
#endif

    const double dbl{std::sqrt(2.25)};
    const int err{errno};
    std::printf("dbl: %g\n", dbl);
    std::printf("err: %d\n", err);
}

C:\GitHub\STL>cl /EHsc /nologo /W4 /std:c++latest /MTd /Od woof.cpp /DUSE_CLASSIC_INCLUDES && woof.exe
woof.cpp
Classic includes.
dbl: 1.5
err: 0

C:\GitHub\STL>cl /EHsc /nologo /W4 /std:c++latest /MTd /Od woof.cpp std.obj && woof.exe
woof.cpp
woof.cpp(19): error C2065: 'errno': undeclared identifier

This happens to be by design (errno is a macro, and named modules don't emit macros). Aside from that, this would be a perfect bug report - it clearly shows that the code works with #include and fails with import std;. It also shows the entire code, command lines, and compiler output.

Bug Bash Issues

Where to Report

• Please report bug bash issues to my personal fork: github.com/StephanTLavavej/STL/issues
• Unlike the official repo, I am unable to accept pull requests to my personal fork (as the CLA Bot doesn't run here). Until this feature merges to official main, please submit issues only.
• If you encounter unrelated issues with the STL, compiler, IDE, etc., please see Reporting Issues in the official repo.
• If you have questions, feel free to join the STL's Discord server and ask in the #modules channel.

Out of Scope

• Mixing named modules and classic includes. This is a known problem that the compiler team is working on. For this bug bash, I'm interested in only pure test cases for named modules with no classic includes, except that <assert.h> should be fine.
• IntelliSense, CMake, MSBuild, etc. This bug bash is for the MSVC toolset only. IntelliSense, powered by the EDG compiler front-end, is actively working on module support but there are major known bugs. Similarly, build system support (CMake and MSBuild) is a work in progress, which is why these examples demonstrate individual command lines. The Clang toolset is not yet ready (but will eventually be supported as a first-class citizen).
• DevCom-1256361 "[C++][Modules][GMF] Named module leaks names from GMF" results in names being exported that shouldn't be. This is a known issue with no library workaround, so any accepts-invalid bugs that this causes are out of scope.
• To deal with 2-arg vs. 3-arg hypot(), I had to export an inline namespace std::_Binary_hypot. This is the only _Ugly identifier that I had to export (aside from compiler bug workarounds).
• The compiler is still working on representing ordinary function bodies in the IFC. (Function templates, and member functions of class templates, are properly represented.) Until this work is completed, there will be a runtime performance difference between classic includes and named modules due to the ability to inline ordinary function bodies at callsites. This also has some correctness impact - the compiler can't enforce /Zc:inline.

In Scope

• Both import std; and import std.compat; should work.
• The named modules should respect all compiler options and other settings that the classic headers respect, including the restoration of removed features. (This is the sole major exception to my otherwise strict policy that the std module should export only Standard names.)
  • My examples depict static debug (/MTd /Od), but the other flavors are certainly within scope (e.g. dynamic release /MD /O2).
  • Similarly, I've depicted building and testing x64 here, but x86 is also within scope, and so is ARM64.
• If I mistakenly exported any other _Ugly identifiers (implicitly or explicitly), those are eligible for accepts-invalid bugs. The named modules should be strict about exporting only publicly-usable machinery.
• You can add /d1ifcInlineFunctions when building the IFC to enable a work-in-progress implementation that addresses the previously mentioned limitation. This appears to work in my limited testing, and it will eventually become the shipping behavior, so I consider it to be in scope for testing.

Scoring

In addition to the satisfaction of helping to bring Standard Library Modules to the world 😻, I will award arbitrary internet points for fun.

• This bug bash will run for 2 weeks, ending at 11:59 PM Pacific Time on Friday, Aug 5, 2022.
• Everyone can report bugs.
• Only bug reports properly submitted as GitHub issues here will be considered. (Accepting reports through other channels would be time-consuming.)
• This bug bash is not an official Microsoft thing, it's just something I'm personally organizing as I work on getting this feature ready.

Scoring system:

• Valid code
  • Accepted
    • Behaves correctly: Yay!
    • Behaves incorrectly: Silent-bad-codegen, 10 points.
  • Rejected
    • Spurious warning: Warning-on-valid, 2 points.
    • Ordinary compiler or linker error: Rejects-valid, 5 points.
    • Internal compiler error: ICE-on-valid, 7 points.
• Bogus code
  • Accepted
    • Accepts-invalid, 3 points.
  • Rejected
    • Correct error: Yay!
    • Bogus error: Wrong-diagnostic, 1 point.
    • Internal compiler error: ICE-on-invalid, 2 points.

Modifiers:

• Double points for library-only bugs (i.e. those where the compiler was working correctly).
• Normal points (no penalties) for bugs which appeared to be unique, but which ultimately shared a root cause.
• Zero points for duplicates of bug bash issues. Please check what's already been found.

Hints

I don't know where the bugs are, or I'd have fixed/reported/worked around them already. 😹 However, I'd recommend exploring areas where the STL has separately compiled components (i.e. the stuff in stl/src like iostreams, multithreading, filesystems, etc.), as getting the named module to work with the STL's DLL, import LIB, and static LIB (which are built as classic non-modularized C++) has been especially challenging. (I need extern "C" and extern "C++" in various places to make this work.)