Process

Table of Contents

• Quick Sheet
• Features
• Status
• Description
• Videos
• Blog
  • Process
  • ProcessChain
  • ProcessEnvironment
  • ProcessFork
• Roadmap

🟩 Create child processes and chain them (also usable with Async library)

Process allows launching, chaining input and output, setting working directory and environment variables of child processes.

Quick Sheet

// 1. Execute child process (launch and wait for it to fully execute)
Process().exec({"cmd-exe", "-h"});
//--------------------------------------------------------------------------
// 2. Execute child process, redirecting stdout to a string
SmallString<256> output; // could be also just String
Process().exec({"where-exe", "winver"}, output);
//--------------------------------------------------------------------------
// 3. Launch a child process and explicitly wait for it to finish execution
Process process;
// This is equivalent to process.exec({"1s", "-l"))
process.launch({"ls", "-l"});
//
// ...
// Here you can do I/0 to and from the spawned process
// ...
process.waitForExitSync();
//--------------------------------------------------------------------------
// 4. Execute child process, filling its stdin with a StringView
// This is equivalent of shell command: `echo "child proc" | grep process`
Process().exec({"grep", "process"}, Process::StdOut::Inherit(), "child proc");
//--------------------------------------------------------------------------
// 5. Read process output using a pipe, using launch + waitForExitSync
Process process5;
PipeDescriptor outputPipe;
process5.launch({"executable.exe", "—argument1", "-argument2"}, outputPipe);
String output5 = StringEncoding::Ascii; // Could also use SmallString<N>
File(outputPipe.readPipe).readUntilEOF(output5);
process5.waitForExitSync(); // call process-getExitStatus() for status code
//--------------------------------------------------------------------------
// 6. Executes two processes piping p1 output to p2 input
Process p1, p2;
ProcessChain chain;
chain.pipe(p1, {"echo", "Salve\nDoctori"});
chain.pipe(p2, {"grep", "Doc"});
// Read the output of the last process in the chain
String output6;
chain.exec(output6);
SC_ASSERT_RELEASE(output == "Doctori\n");
//--------------------------------------------------------------------------
// 7. Set an environment var and current directory for child process 
Process process7;
// This child process7 will inherit parent environment variables plus NewEnvVar
SC_TEST_EXPECT(process7.setEnvironment("NewEnvVar", "SomeValue"));
// This child process7 will inherit parent environment variables but we re-define PATH
SC_TEST_EXPECT(process7.setEnvironment("PATH", "/usr/sane_cpp_binaries"));
// Set the current working directory
SC_TEST_EXPECT(process7.setWorkingDirectory("/usr/home"));

Features

Class	Description
SC::Process	Execute a child process with standard file descriptors redirection.
SC::ProcessChain	Execute multiple child processes chaining input / output between them.
SC::ProcessEnvironment	Reads current process environment variables.
SC::ProcessFork	Forks current process exiting child at end of process A fork duplicates a parent process execution state, os handles and private memory.

Status

🟩 Usable
Library is being used in SC::Plugin and in SC::Tools.

Description

The SC::Process class is used when handling a process in isolation, while the SC::ProcessChain is used when there is need to chain inputs and outputs of multiple processes together.

Videos

This is the list of videos that have been recorded showing some of the internal thoughts that have been going into this library:

• Ep.06 - Posix fork

Blog

Some relevant blog posts are:

• March 2024 Update
• April 2024 Update
• April 2025 Update

Process

Execute a child process with standard file descriptors redirection.

Features:

• Redirect standard in/out/err of a child process to a Pipe
• Inherit child process file descriptors from parent process
• Ignore (silence) child process standard file descriptor
• Wait for the child process exit code

Example: execute child process (launch and wait for it to fully execute)

    // Example: execute child process (launch and wait for it to fully execute)
    SC_TRY(Process().exec({"cmd.exe", "-h"}));

Example: execute child process, redirecting stdout to a string

    // Example: execute child process, redirecting stdout to a string
    SmallString<256> output; // could be also just String
    SC_TRY(Process().exec({"where.exe", "winver"}, output));
    // Output now contains "C:\Windows\System32\winver.exe\n"

Example: launch a child process and wait for it to finish execution

    // Example: launch a child process and explicitly wait for it to finish execution
    Process process;
    SC_TRY(process.launch({"ls", "-l"}));
    // ...
    // Here you can do I/O to and from the spawned process
    // ...
    SC_TRY(process.waitForExitSync());
    // This is equivalent to process.exec({"ls", "-l"})

Example: execute child process, filling its stdin with a StringView

    // Example: execute child process, filling its stdin with a StringView
    // This is equivalent of shell command:
    // `echo "child process" | grep process`
    SC_TRY(Process().exec({"grep", "process"}, Process::StdOut::Inherit(), "child proc"));

Example: read process output using a pipe, using launch + waitForExitSync

    // Example: read process output using a pipe, using launch + waitForExitSync
    Process        process;
    PipeDescriptor outputPipe;
    SC_TRY(process.launch({"executable.exe", "--argument1", "--argument2"}, outputPipe));
    String output = StringEncoding::Ascii; // Could also use SmallString<N>
    SC_TRY(File(outputPipe.readPipe).readUntilEOF(output));
    SC_TRY(process.waitForExitSync());
    // ... Do something with the 'output' string

Example: Add an environment variable

    Process process;
    // This child process will inherit parent environment variables plus NewEnvVar
    SC_TEST_EXPECT(process.setEnvironment("NewEnvVar", "SomeValue"));
    String output;
    // Spawn the child process writing all env variables as KEY=VALUE\n to stdout, redirected to output
    SC_TEST_EXPECT(spawnChildAndPrintEnvironmentVars(process, output));
    // We can check that the NewEnvVar has been set to SomeValue
    SC_TEST_EXPECT(output.view().containsString("NewEnvVar=SomeValue"));
    // PATH env var exists because we are inheriting environment
    SC_TEST_EXPECT(output.view().containsString("PATH="));

Example: Redefine an environment variable

    Process process;
    // This child process will inherit parent environment variables but we re-define PATH
    SC_TEST_EXPECT(process.setEnvironment("PATH", "/usr/sane_cpp_binaries"));
    String output;
    // Spawn the child process writing all env variables as KEY=VALUE\n to stdout, redirected to output
    SC_TEST_EXPECT(spawnChildAndPrintEnvironmentVars(process, output));
    // PATH env var has been re-defined
    SC_TEST_EXPECT(output.view().containsString("PATH=/usr/sane_cpp_binaries"));

Example: Disable environment variable inheritance

    Process process;
    process.inheritParentEnvironmentVariables(false);
    String output;
    // Spawn the child process writing all env variables as KEY=VALUE\n to stdout, redirected to output
    SC_TEST_EXPECT(spawnChildAndPrintEnvironmentVars(process, output));
    // PATH env var doesn't exist because of Process::inheritParentEnvironmentVariables(false)
    SC_TEST_EXPECT(not output.view().containsString("PATH="));

ProcessChain

Execute multiple child processes chaining input / output between them.

Chains multiple child processes together, so that the output of a process becomes input of another (similar to what happens wit the pipe (|) operator on Posix shells).

SC::PipeDescriptor from File library is used to chain read / write endpoints of different processes together.

Example: Inherit stdout file descriptor

    // Executes two processes piping output of process p1 to input of process p2.
    // Then reads the output of the last process in the chain and check its correctness.
    ProcessChain chain;
    Process      p1, p2;
    // Print "Salve\nDoctori" on Windows and Posix and then grep for "Doc"
    StringView expectedOutput;
    switch (HostPlatform)
    {
    case Platform::Windows: {
        expectedOutput = "Doctori\r\n";
        SC_TEST_EXPECT(chain.pipe(p1, {"cmd", "/C", "echo", "Salve", "&", "echo", "Doctori"}));
        SC_TEST_EXPECT(chain.pipe(p2, {"findstr", "Doc"}));
    }
    break;
    default: { // Posix
        expectedOutput = "Doctori\n";
        SC_TEST_EXPECT(chain.pipe(p1, {"echo", "Salve\nDoctori"}));
        SC_TEST_EXPECT(chain.pipe(p2, {"grep", "Doc"}));
    }
    break;
    }
    String output;
    SC_TEST_EXPECT(chain.exec(output));
    SC_TEST_EXPECT(output == expectedOutput);

Example: Read stderr and stdout into a string

    // Executes two processes piping output of process p1 to input of process p2.
    // Reads p2 stdout and stderr into a pair of Strings.
    ProcessChain chain;
    Process      p1;
    StringView   expectedOutput;
    switch (HostPlatform)
    {
    case Platform::Windows: {
        expectedOutput = "C:\\Windows\\System32\\where.exe\r\n";
        SC_TEST_EXPECT(chain.pipe(p1, {"where", "where.exe"}));
    }
    break;
    default: { // Posix
        expectedOutput = "DOCTORI\n";
        SC_TEST_EXPECT(chain.pipe(p1, {"echo", "DOCTORI"}));
    }
    break;
    }
 
    String stdOut(StringEncoding::Ascii);
    String stdErr(StringEncoding::Ascii);
    SC_TEST_EXPECT(chain.exec(stdOut, Process::StdIn::Inherit(), stdErr));
    SC_TEST_EXPECT(stdOut == expectedOutput);
    SC_TEST_EXPECT(stdErr.isEmpty());

Example: Read standard output into a string using a Pipe

    // Chain two processes and read the last stdout into a String (using a pipe)
    ProcessChain chain;
 
    String  output(StringEncoding::Ascii);
    Process p1, p2;
 
    StringView expectedOutput;
    switch (HostPlatform)
    {
    case Platform::Windows: {
        expectedOutput = "WHERE [/R dir] [/Q] [/F] [/T] pattern...\r\n";
        SC_TEST_EXPECT(chain.pipe(p1, {"where", "/?"}));
        SC_TEST_EXPECT(chain.pipe(p2, {"findstr", "dir]"}));
    }
    break;
    default: { // Posix
        expectedOutput = "sbin\n";
        SC_TEST_EXPECT(chain.pipe(p1, {"ls", "/"}));
        SC_TEST_EXPECT(chain.pipe(p2, {"grep", "sbin"}));
    }
    break;
    }
    PipeDescriptor outputPipe;
    SC_TEST_EXPECT(chain.launch(outputPipe));
    SC_TEST_EXPECT(File(outputPipe.readPipe).readUntilEOF(output));
    SC_TEST_EXPECT(chain.waitForExitSync());
    SC_TEST_EXPECT(output.view().startsWith(expectedOutput));

ProcessEnvironment

Reads current process environment variables.

Example: Print all environment variables to stdout

    SC::ProcessEnvironment environment;
    for (size_t idx = 0; idx < environment.size(); ++idx)
    {
        StringView name, value;
        (void)environment.get(idx, name, value);
        if (value.isEmpty())
        {
            report.console.printLine(name);
        }
        else
        {
            report.console.print(name);
            report.console.print("=");
            report.console.printLine(value);
        }
    }

ProcessFork

Forks current process exiting child at end of process A fork duplicates a parent process execution state, os handles and private memory.

Its semantics are quite different from platform to platform but on its most common denominator it can be used to carry on "background" operations on snapshots of current program memory. One relevant use case is serializing to disk or network a live, complex and large data structure. Without the fork the program should either:

1. Duplicate all the data, to snapshot it in a given instant, and keep it around for Async IO
2. Block program execution and write the live data-structure until all IO is finished

Fork avoids memory duplication because it will be shared through Copy On Write (COW) mechanisms. COW ensures that un-modified duplicated memory pages will not occupy additional Physical RAM.

A pair of pipes makes it easy to do some coordination between parent and forked process.

Warning

There are really MANY caveats when forking that one should be aware of:

1. Many API will just not work as expected on the forked process, especially on Windows
2. Limit API calls in forked process to console IO, network and file I/O (avoid GUI / Graphics)
3. All threads other than the current one will be suspended in child process (beware of deadlocks)
4. Create Sockets and FileDescriptors with Inheritable flags if you need them in fork process
5. Process deadlocks under Windows ARM64 / x86 emulation (use Process::IsWindowsEmulatedProcess)

Example: Fork current process modifying memory in forked process leaving parent's one unmodified.

    // Cross-platform lightweight clone of current process, sharing memory
    // but keeping any modification after clone "private" (Copy-On-Write).
    // Achieved using "fork" on Posix and "RtlCloneUserProcess" on Windows.
    StringView sharedTag = "INITIAL";
    StringView parentTag = "PARENT";
    StringView saveFile  = "ForkSaveFile.txt";
 
    // The string will be duplicated using Copy-On-Write (COW)
    String shared = sharedTag;
 
    // CLONE current process, starting child fork in Suspended state
    // Forked process will be terminated by ProcessFork destructor
    ProcessFork fork;
    SC_TEST_EXPECT(fork.fork(ProcessFork::Suspended));
 
    // After fork program must check if it's on fork or parent side
    switch (fork.getSide())
    {
    case ProcessFork::ForkChild: {
        report.console.printLine("FORKED process");
        report.console.print("FORKED Shared={0}\n", shared);
 
        // Write the "shared" memory snapshot to the file system
        FileSystem fs;
        SC_TEST_EXPECT(fs.init(report.applicationRootDirectory));
        SC_TEST_EXPECT(fs.writeString(saveFile, shared.view()));
 
        // Send (as a signal) modified string contents back to Parent
        SC_TEST_EXPECT(fork.getWritePipe().write({shared.view().toCharSpan()}));
    }
    break;
    case ProcessFork::ForkParent: {
        report.console.printLine("PARENT process");
        // Check initial state to be "INITIAL" and modify shared = "PARENT"
        report.console.print("PARENT Shared={0}\n", shared);
        SC_TEST_EXPECT(shared == sharedTag and "PARENT");
        shared = parentTag;
 
        // Resume suspended fork verifying that on its side shared == "INITIAL"
        SC_TEST_EXPECT(fork.resumeChildFork());
        char       string[255] = {0};
        Span<char> received;
        SC_TEST_EXPECT(fork.getReadPipe().read(string, received));
        StringView stringFromFork(received, true, StringEncoding::Ascii);
        report.console.print("PARENT received={0}\n", stringFromFork);
        SC_TEST_EXPECT(stringFromFork == sharedTag);
 
        // Check creation of "save file" by fork and verify its content too
        FileSystem fs;
        SC_TEST_EXPECT(fs.init(report.applicationRootDirectory));
        String savedData;
        SC_TEST_EXPECT(fs.read(saveFile, savedData, StringEncoding::Ascii));
        SC_TEST_EXPECT(savedData == sharedTag);
        SC_TEST_EXPECT(fs.removeFile(saveFile));
 
        // Optionally wait for child process to exit and check its status
        SC_TEST_EXPECT(fork.waitForChild());
        SC_TEST_EXPECT(fork.getExitStatus() == 0);
    }
    break;
    }

Roadmap

🟦 Complete Features:

• To be defined

💡 Unplanned Features:

• None so far