☔️ Sane C++ March 24

SC::Tools#

written for Sane C++ Libraries, using Sane C++ Libraries

There is no better way to improve a library than using it to create some other project or tool.
This is why effort has been spent using Sane C++ Libraries to build some useful tools needed by...Sane C++ Libraries!

So, unsurprisingly, the big addition of the month to Sane C++ Libraries are SC::Tools!

Something similar to SC::Tools already used to exist in some form: SCBuild.cpp, the bootstrap from source C++ file using SC::Build build system (generator).
It used to work thanks to a bootstrap bash / batch file that compiles SC-Build.cpp and the unity build file SC.cpp to produce the SCBuild[.exe] self-contained (statically linked) executable.
This was nice but it had a few issues, for example it was not able to detect changes in any of the source files and rebuild the executable accordingly.

SC::Tools is a generalization of the SCBuild.{sh | bat} mechanism into a more flexible SC.sh $TOOL [$ACTION]. This allows also running custom tools, leveraging Sane C++ Libraries, by just launching SC.sh myDirectory/myScript.cpp [$ACTION]!

The source code dependency tracking is taken care of by a couple of parametric makefiles.
The intermediate files directory is shared by all tools, and the unity build file SC.cpp (#include-d by Tools.cpp ) makes the entire set of Sane C++ Libraries available to every tool.
It's compiled just once (in a couple of seconds), making any successive compile / linking of the tools extremely fast (less than 1 sec on a recent laptop).

Tools in theTools subdirectory are special, as they can be invoked by just using their name (without the SC- prefix).
./SC.sh Tools/SC-build.cpp can be invoked as just ./SC.sh build.

SC-build.cpp#

building C++ programs in C++, why not?

The new action called compile (guess what?) compiles the generated projects 🤯!

./SC.sh Tools/SC-build.cpp configure
or
./SC.sh build configure

and on windows:

SC.bat Tools\SC-build.cpp compile
or
SC.bat build compile

SC-package.cpp#

wheel reinvention, reinvented

This poor-man's imperative package manager downloads binary files taking care of checking MD5, extracting and testing what's contained.
The guiding reason for this tool to exist was to automatically be able downloading and extracting the officialclang distribution (at a specific version, matching the Gitlab CI) in order to use clang-format on the entire repo.
This is needed for a potential contributor to properly format files before sending a Pull Request, keeping the CI format validation jobs happy.

The end result is a folder with _PackagesCache and _Packages under _Build.

This is still very primitive for now, but who knows, maybe in the future it could be evolved into something more advanced.

Additional reasons to exist#

other than wheel re-invention

Too often repository automation tools are written inbash or batch where best thing you can do is putting print (echo) statements to debug them.
Slightly better are the cases when an actual programming language like python or javascript is used, but sometimes this comes with an entire set of problems regarding their requirements and in general the fact that such tools expect to be installed globally on developer's system, with probability of incurring into a it works on my machine situation.
Things like Python .venv can sometimes help but not always.
In some cases such scripts do not even exist, but you're getting a bunch of install this or install that Readmes always missing some step and where you end up wasting a lot of time.
And let's not discuss docker about please, I am trying to stay nice 😄.

So my obvious solution has been to write such a tool in C++ using Sane C++ Libraries of course, so that I can debug it properly just like any other program.
I am really happy the result, the small program is easy to follow and does its job.
For now all data is local to the repository _Build directory but it would be easy sharing the _PackagesCache into some custom user directory to save some space on multiple checked-out copies of the repository.
For any issue, I can just debug this C++ shell script, that is just a regular program, like all other programs that a C++ programmer debugs every day.

Technical Details#

if you really want to know...

On Posix to getclang+llvm-x.y.z-$ARCH-$PLATFORM you just download a file from github, check it MD5, and extract the .tar.gz file.

On Windows this is even more difficult because the official LLVM binary distribution is some sort of self-extracting setup (requiring admin privileges too!) and to extract it you need a tool like 7zip. 7zip is as well a self extracting installer and requires another tool (called 7zr.exe) in order to be extracted!. So on Windows SC-package downloads 7zr.exe -> extracts the 7zip installer -> uses the extracted 7z.exe to extract the LLVM-X.Y.Z-$ARCH into some directory. If someone knows an easier way of doing this (without installing other tools or requiring a package manger) please let me know.

On all platforms / architectures (macOS/Windows/Linux and ARM64/Intel64) MD5 of the downloaded files it's being checked to make sure they're all legit. This also helps creating a _PackagesCache to hold these setup files and avoid re-downloading them every time.

Also a small clang-format test is run to make sure it's properly installed and available. The extracted directory is symlinked into a _Packages sub-folder containing package name / platform.

Sane C++ Libraries doesn't have (yet) a properly working HTTPS client, so an alternative solution was necessary. It looks like nowadayscurl is pre-installed on any recent macOS,linux andwindows install and so SC-package uses it.

SC-format.cpp#

code is poetry, prove me wrong

It launches many parallel processes but always dynamically limiting them to the number of available processors using SC::Process. Processes are monitored using the SC::Async Library, and as soon as one "slot" is made available, it's being filled with a new clang-format process.

Most code has upper bounds and fixed buffers.
Even if it's totally unnecessary for such a small script, it's cool to see how the libraries work together to allow minimal (or no) heap allocation!
In this example the number of maximum processes is fixed and the recursive file system iteration happens with a lambda, avoiding storing a vector of all file paths found in the target directory.

The tool looks for clang-format in known places, depending on platform, and if a working clang-format at the correct version is found, it will be used. This is for example what happens on CI as clang-format is pre-installed on Github CI OS Images. If the no suitable clang-format has been found, then SC-format.cpp relies on SC-package.cpp to download a brand new set of LLVM binaries from github, extracting and symlinking them.

Improvements to the CI and .vscode files#

These are the new simplified CI and VSCode build scripts:

Improvements to the Library#

SC::Process

The original interface was not so easy to use so it got a nice facelift:

• Process: Improve the API to be more pleasant to use

It also got some more features:

• Process: Allow ignoring child process standard output / error
• Process: Add Process::getNumberOfProcessors

Improved Tests and Documentation:

• Process: Cleanup tests and improve documentation

and bugfixes for very tricky OS-specific issues!

• Process: Fix Posix waitForExitSync failing on non zero exit status
• Process: Handle kevent ESRCH errors caused by processes exiting too fast

SC::Build#

As already said the SC::Build library got the capability of compiling specific configuration / architecture combos.

This has been also the occasion to cleanup the intermediates / output folder naming, so that all folders can coexist in the same working directory. This is useful to map that very same working directory on all Operating Systems (with a couple of Virtual Machines), so that code can be quickly tested on all platforms while it's being developed, before even hitting the CI.

Features:

• Build: Support compiling specific configuration / architecture

SC::Async#

The SC::Async library got some fixes in the child process exit handling, that was randomly failing on theSC-format command!

Features:

• Async: Remove the handle from active list before calling complete callback
  This allows reusing "completed" AsyncRequests in the callback for a new operation!)

Bugfixes:

• Async: Handle merged SIGCHLD by signalfd on epoll backend
  This was causing spuriously missing child process exit notifications, and it was only visible on Linux (Intel) with a "high enough" number of concurrent process finishing "simultaneously".

SC::FileSystem#

SC::FileSystem got some additions needed mainly by SC-package and a bugfix found when developing the tool.

Features:

• FileSystem: Add existsAndIsLink, removeLinkIfExists and moveDirectory
• FileSystem: Add createSymbolicLink
• File: Add read / write uint8_t overloads and make readAppend private

Bugfixes:

• File: Pop null terminator before appending to String with readUntilEOF

SC::Strings#

Features:

• Strings: Add StringView {starts | ends}WithAnyOf and trim{ start | end }AnyOf
• Strings: Allow usingSmallString as argument to format
• Strings: Add casing parameter to StringBuilder::appendHex
• Strings: Add Windows only UTF16 StringView::fromNullTerminated
• Strings: Add StringFormatterFor void* pointers

Bugfixes:

• Strings: Fix an UB when copying StringView containing null