Start-Process -UseNewEnvironment provides an environment that is missing crucial standard environment variables while not supporting passing a new environment

[mklement0]

On both Windows and Unix platforms, Start-Process -UseNewEnvironment results in an environment that is missing crucial standard environment variables, making the new environment virtually useless, while not providing a mechanism to define a new environment:

• On Windows, -UseNewEnvironment defines only the variables that are explicitly defined, as displayed in System Properties (sysdm.cpl), with crucial, usually automatically defined variables such as $env:ProgramFiles missing, and $env:USERNAME unexpectedly containing SYSTEM.

  • This even prevents another powershell instance from being started this way - see Start-Process fails to launch new powershell.exe instance with -UseNewEnvironment switch #3545 (comment) and below.

• On Unix, an empty environment is essentially passed (with no environment variables defined at all, except $env:PSModulePath).

  • This even prevents invoking any executable by filename only, given that $env:PATH is undefined - see below.

  • While this behavior is similar to POSIX env -i on Unix platforms, the env utility notably includes a mechanism to provide a new environment via name=value pairs, which Start-Process lacks.

---

Possible solutions:

• Repurpose -UseNewEnvironment to not start with a blank slate / crucial variables missing, but to provide the same environment that a shell would see when directly launched by the system (in other words: any environment-variable modifications made by the calling shell would be ignored).

• Additionally, provide a way to pass a custom environment, applied on top of the current or pristine (-UseNewEnvironment) environment:

  • E.g., a new -Environment <hashtable> / -Environment <Collections.DictionaryEntry[]> parameter could be used.

  • If someone truly wanted an empty environment, they could start with $emptyEnv = (Get-ChildItem env:); $emptyEnv | % { $_.Value = $null } and pass -Environment $emptyEnv.

---

The -Environment feature would allow for an - incomplete - approximation of the convenient ad-hoc, command-scoped environment-variable definition feature that POSIX-like shells such as bash offer, where you can prepend one or more name=value pairs to a command:

# Bash (any POSIX-compatible shell)
# Defines $env:FOO as 'bar', but *only for the some-utility child process*.
FOO=bar some-utility 666

Update: #3316 suggests emulating this syntax in PowerShell, which would be the best solution.

The PS approximation would be:

Start-Process -Wait  -Environment @{ FOO = 'bar' } some-utility -Args 666

That said, a crucial limitation is that use of Start-Process makes the external utility operate outside PowerShell's streams, so the only way to provide input / collect output is via the -Redirect* parameters, which requires auxiliary files.

Current behavior

Start-Process -UseNewEnvironment -Wait -NoNewWindow pwsh -args '-Command', 'gci env:; whoami'

• On Windows: PowerShell refuses to start - see Start-Process fails to launch new powershell.exe instance with -UseNewEnvironment switch #3545; note that adding -LoadUserProfile makes no difference.

Internal Windows PowerShell error.  Loading managed Windows PowerShell failed with error 8009001d.

• On Unix: PowerShell starts, lists $env:PSModulePath as the only environment variable - with a seemingly temporary user account's module directory prepended - and the whoami invocation fails, because it cannot be located in the absence of a suitable $env:PATH.

Name                           Value                                                                                                                           
----                           -----                                                                                                                           
PSModulePath                   /tmp/ba38e79f-40c2-440e-ae08-7cf32e0708e1/.local/share/powershell/Modules:/usr/local/share/powershell/Modules:/opt/microsoft/...
whoami : The term 'whoami' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path 
was included, verify that the path is correct and try again.
At line:1 char:21
+ Get-ChildItem env:; whoami
+                     ~~~~~~
    + CategoryInfo          : ObjectNotFound: (whoami:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

Environment data

PowerShell Core v6.0.0-beta.5 on macOS 10.12.6
PowerShell Core v6.0.0-beta.5 on Ubuntu 16.04.3 LTS
PowerShell Core v6.0.0-beta.5 on Microsoft Windows 10 Pro (64-bit; v10.0.15063)
Windows PowerShell v5.1.15063.483 on Microsoft Windows 10 Pro (64-bit; v10.0.15063)

[GeeLaw]

On Windows, you should use CreateEnvironmentBlock function to create a complete block of environment variables for a specified user.

There is another issue on UseNewEnvironment. When combined with Credential, it currently starts the subprocess in the new user's context, but with the current user's environment variables, which is surely a bad thing to do. If instead, you turn off UseNewEnvironment and supply Credential, the subprocess will use the default environment block for the specified user, which is likely what the user wants.

[mklement0]

To provide some more insight into what, specifically, is wrong (in addition to the conceptual flaw that @GeeLaw mentions with respect to combining -UseNewEnvironment with -Credential; almost sounds like that combination should be disallowed):

PowerShell/src/Microsoft.PowerShell.Commands.Management/commands/management/Process.cs

Lines 1867 to 1872 in 5fcab21

	if (_UseNewEnvironment)
	{
	startInfo.EnvironmentVariables.Clear();
	LoadEnvironmentVariable(startInfo, Environment.GetEnvironmentVariables(EnvironmentVariableTarget.Machine));
	LoadEnvironmentVariable(startInfo, Environment.GetEnvironmentVariables(EnvironmentVariableTarget.User));
	}

On Unix-like platforms, the LoadEnvironmentVariable() calls are effective no-ops, because Environment.GetEnvironmentVariables(EnvironmentVariableTarget.Machine) and Environment.GetEnvironmentVariables(EnvironmentVariableTarget.User) quietly(!) return an empty hashtable in Unix, where these targets aren't supported (see https://github.com/dotnet/corefx/issues/32685).

In short, LoadEnvironmentVariable() shouldn't even be called to begin with on non-Windows platforms.

As for Windows:

• The original post mentions that crucial built-in variables are missing from the resulting process, which results from the approach of first clearing all variables with

PowerShell/src/Microsoft.PowerShell.Commands.Management/commands/management/Process.cs

Line 1869 in 5fcab21

startInfo.EnvironmentVariables.Clear();

and then only restoring those variables whose definitions are registry-based.
However, there are a number of crucial variables such as %USERPROFILE% that are not registry-based, so they end up missing altogether - see below.

• Also, loading a default $env:PATH value is broken in LoadEnvironmentVariable(), because key access is case-sensitive and the variable name is mistakenly tested for with all-uppercase name PATH:

PowerShell/src/Microsoft.PowerShell.Commands.Management/commands/management/Process.cs

Line 2077 in 5fcab21

if (entry.Key.ToString().Equals("PATH"))

---

For the record, here are the built-in, system-defined environment variables that are not registry-based:

ALLUSERSPROFILE
APPDATA
CommonProgramFiles
CommonProgramFiles(x86)
CommonProgramW6432
COMPUTERNAME
HOMEDRIVE
HOMEPATH
LOCALAPPDATA
LOGONSERVER
ProgramData
ProgramFiles
ProgramFiles(x86)
ProgramW6432
PUBLIC
SESSIONNAME
SystemDrive
SystemRoot
USERNAME    # !! There is a *machine-level* definition with value 'SYSTEM', but there is no *user-level* definition - that is added by the system to the process.
USERDOMAIN
USERDOMAIN_ROAMINGPROFILE
USERPROFILE

Command that generated the list above (with the exception of USERNAME)

gci env: | % name | ? { -not [environment]::GetEnvironmentVariable($_, 'Machine') -and -not [environment]::GetEnvironmentVariable($_, 'User')}

[FranklinYu]

I put the "possible solution 2" in #8352. Even if this issue is fixed by implementing solution 1, I believe solution 2 has other use-cases.

Update: that's a dupe of #3316.

[iSazonov]

I'd deprecate UseNewEnvironment (make alias?) and introduce ClearEnvironment
I like new Environment parameter but can we explicitly say that we add/change variable? Maybe -SetEnvironmentVariable?

[SteveL-MSFT]

On GitHub, there's a small amount of usage of -UseNewEnvironment. I believe the intent of this switch is you would start a process from PowerShell as though you just started a new PowerShell process (not from the current PowerShell process). We just have to evaluate if we would break any existing usage (seems unlikely).

As for specifying an environment, I think -Environment @{} is good.

[mklement0]

Agreed re intent, @SteveL-MSFT, so if we wanted to fix this feature (and also make it work on Unix), we could cache a copy of the environment-variable block on PowerShell startup (before reading profiles), and use that.

@iSazonov: I agree that -SetEnvironmentVariable is more descriptive, but it's a bit unwieldy. I think with proper documentation -Environment will do, as Steve says.

-ClearEnvironment wouldn't quite capture the intent of the feature, which mustn't clear all variables - that it currently does so on Unix is what prompted this report.

[iSazonov]

I think really we need three operation:

1. ClearEnvironment [optional hash with new variables to set]
2. RemoveEnvironment
3. SetEnvironment <mandatory hash with variables to add/update to current environment>

UseNewEnvironment we could deprecate and hide.

[mklement0]

I don't think there is a need for this complexity:

Re 1 and 2:

You rarely want to start with no environment variables at all - you'll wipe out vital OS-defined variables too (which is the current problem on Unix-like platforms).

It is far more common to build on an existing environment, either:

• the current one

• "pristine" one, without any in-session changes, but with the usual OS-defined ones present - that's what -UseNewEnvironment should do.

Re 3:

Combining the hashtable -Environment parameter with -UseNewEnvironment then covers all use cases:

As a user, I then:

• decide which environment to start with: the current one (no switch) or a "pristine" one (-UseNewEnvironment)

• then use -Environment to modify that starting environment, by:

  • adding new variables
  • modifying the values of existing variables
  • removing variables

The first 2 modifications are most common, but a single, hashtable-based environment variable also supports removing variables, given that assigning $null or the empty string to environment variables deletes them; e.g., -Environment @{ FOO='bar'; BAR=$null } would create or update env. var FOO and remove (or not define) env. var. BAR

[iSazonov]

I don't think there is a need for this complexity:

Historically environment variables is manipulated by one "Set" operation, even in C# API https://docs.microsoft.com/en-us/dotnet/api/system.environment.setenvironmentvariable?view=netframework-4.8#System_Environment_SetEnvironmentVariable_System_String_System_String_
But a question is can we get PowerShell magic benefits having explicit operations like *-Variables?

CommandType     Name 
-----------     ----
Cmdlet          Clear-Variable
Cmdlet          Get-Variable
Cmdlet          New-Variable
Cmdlet          Remove-Variable
Cmdlet          Set-Variable

Earlier we actively discussed Environment provider/drive. We found this very interesting. Obviously the API will has these operations explicitly. Should we keep consistency with the API in the issue too (also taking in account hosting scenario)?

I do not like changing UseNewEnvironment for 4 reasons: unlikely but breaking change, the name confuses me, not clear why we need the rare scenario and implementation will slow down startup that is very bad. I believe this is not worth the effort until there is a real important business scenario. Moreover, PowerShell actually only changes PSModulePath, and this can be solved in #10300.

Environment.SetEnvironmentVariable Method (System)

Creates, modifies, or deletes an environment variable.

[mklement0]

The *-Variable cmdlets are narrowly focused on shell variables (PowerShell's own, in-session only variables).

However, extending the Environment PS provider to also support managing persisted environment variables, at least on Windows, makes perfect sense to me, and it is what I've proposed here.

I don't see a problem with having that coexist with the -Environment <hashtable> parameter proposed here, or even with the syntax discussed in #3316.

I do not like changing UseNewEnvironment for 4 reasons: unlikely but breaking change

Nothing meaningful can be broken, given the currently useless behavior.

the name confuses me

I think the name is fine, but we could implement an alias, if there's consensus on a better name.

not clear why we need the rare scenario and implementation will slow down startup that is very bad

That is a valid concern - I can't speak to the real-world impact.

I believe this is not worth the effort until there is a real important business scenario.

Personally, I won't miss the feature, but I can see how it is desirable in certain situations.

Moreover, PowerShell actually only changes PSModulePath

That's not the point, though. The point is that you yourself or third-party code may have modified the environment in multiple ways that you don't want external processes to see, lest it interfere with their proper operation.

[iSazonov]

That's not the point, though. The point is that you yourself or third-party code may have modified the environment in multiple ways that you don't want external processes to see, lest it interfere with their proper operation.

Interesting, has Windows or Unix an API to directly implement the scenario? (I mean this is too unbelievable scenario)

[mklement0]

Windows has Start-Process -UseNewEnvironment :) - though it is partially broken, as I've pointed out. However, I've always understood the intent of that feature to be what I've proposed.

Unix has the env utility with the -i option (ignore the inherited environment, and only define the variables passed as the other arguments);

On closer examination, however, this appears to be more like a -ClearEnvironment switch, and the only reason it results in a usable shell if you call bash is that bash - unlike PowerShell - apparently sets default values for variables such as $env:PATH.

As a cross-platform shell, however, we'd be looking for standardized behavior, so IF we keep the switch, implementing the proposed behavior makes the most sense to me.

Again, I personally wouldn't miss -UseNewEnvironment, and I can see that it is probably rarely needed overall (as at least historically indicated by @SteveL-MSFT's GitHub search).
I think that's all I have to say on this particular aspect of this debate, and I'll let others decide.

[iSazonov]

It seems UseNewEnvironment doesn't work on Unix. 😕

[mklement0]

@iSazonov: Well, like env -i, I think it radically clears the environment, though that is accidental behavior (the current attempts to read persisted env.-var definitions are quiet no-ops on Unix-like platforms, as explained above).

However, there's more going on, because env -i bash --norc results in a functional shell, whereas the seemingly equivalent Start-Process -UseNewEnvironment bash --norc exhibits very strange behavior with the default terminal apps on macOS 10.14 and and Ubuntu 18.04.

Only with iTerm on macOS could I get Start-Process -UseNewEnvironment bash --norc to work, and then call export to see what env. variables are defined: you'll see that they're only Bash-defined ones (as with env -i bash --norc), implying that an empty environment was passed, and Bash seemingly furthermore declares shell-only variables (you can see these among the output from set) of the same name as vital OS env. variables such as PATH and TERM so as to provide at least a minimal level of functioning.

On both platforms, trying to launch pwsh this way always results in broken behavior, because PowerShell does not anticipate and compensate for the absence of vital environment variables.

[iSazonov]

This code is This code is somewhat confusing (in CoreFX too) 😕

This works in WSL (may be useful for tests):

Start-Process -UseNewEnvironment -ArgumentList "set" bash

[iSazonov]

We need to take in account #6489 Again slow down :-(
If we look Process.ProcessStartInfo.Environment - it force us to copy Dictionary and it again slow down.

This is not so important to start a process but it can be sensitive for startup scenario.

[mklement0]

As for the test command: You're missing -c, and it's better to also suppress loading of .bashrc with --norc; set shows you a mix of shell-only and environment variables, so it's better to use export; thus:

Start-Process  -UseNewEnvironment  -ArgumentList '--norc', '-c', 'export'  bash

If you're experimenting with passing multiple commands to -c, remember the Start-Process bug with respect to spaces (#5576), so embedded quoting is needed:

Start-Process  -UseNewEnvironment  -ArgumentList '--norc', '-c', '"export; set"'  bash

[mklement0]

Also, I don't think we need to worry about #6489 for simply making a copy of the startup environment.

As for the dictionary: are you referring to [environment]::GetEnvironmentVariables()? [System.Diagnostics.Process]::GetCurrentProcess().StartInfo is $null in a PowerShell session.

What [environment]::GetEnvironmentVariables() returns already is a copy of the environment, returned as a case-sensitive [hashtable] (IDictionary) whose later manipulation has no effect on the underlying environment variables.

[iSazonov]

Also, I don't think we need to worry about #6489 for simply making a copy of the startup environment.

Perhaps we can catch a problem if original set contains duplicate variables: we have to set them one-by-one and last will win. Although we use P/Invoke on Windows and this could allow to get around this.

[System.Diagnostics.Process]::GetCurrentProcess().StartInfo is $null in a PowerShell session.

It is not property in Process object. So we can use only [environment]::GetEnvironmentVariables().

Start-Process -UseNewEnvironment -ArgumentList '--norc', '-c', 'export' bash

It does not work for me on WSL. Folow works:
Start-Process -UseNewEnvironment -ArgumentList '--norc', '-c', 'printenv' bash

I hope we can use this on all Unix-s and MacOs.

[mklement0]

Yes, printenv is also preinstalled on macOS (it isn't a POSIX-mandated utility, however), so this should work; I'm baffled that export wouldn't work, though, given that it is a Bash builtin (an internal command).

[mklement0]

I just discovered the cmd.exe-internal start command's /I switch, which was presumably the model for PowerShell's -UseNewEnvironment switch (in intent, not in effect, due to the flawed current implementation); from start /? (lightly edited):

/ I ... The new environment will be the original environment passed
to the cmd.exe and not the current environment.

[iSazonov]

@mklement0 Thanks! I pulled PR - please review.

[iSazonov]

Since we had PowerShell Committee conclusion in #10745 (comment) we can continue here to discuss a desired behavior.
From my understanding the conclusion was to implement on all platforms behavior like @rjmholt made for login shell on Unix - with UseNewEnvironment subprocess gets environment like we get with sh -l on Linux and zsh -l on macOS. I don't know can we do the same on Windows.
@SteveL-MSFT @mklement0 Thoughts?

[mklement0]

Thanks, @iSazonov; I think you meant to link to #10745 (comment) - given that specific points were made there, let me respond there.

[SteveL-MSFT]

@iSazonov on Windows, the current code clears the env block, then fills it from the system profile (hence username is system) and then the user profile overwriting any that exist in the system. Hence Path is incomplete. It seems the fix here is to merge Path (append user PATH to end of machine PATH). USERNAME should be special cased to the user. The test is to start a new cmd.exe from Windows shell, run set and have the env vars match if using -UseNewEnvironment.

[SteveALee]

A slight;y different use case is for -UseNewEnvironment to use the default environment when somehting like the path has changed in another procerss.

Specifically I'm using curl to download python and the script to initiate the install and want to start a shell with python on the path.

My original script is windows cmd but as there was no way to do this I looked at PowerShell and got excited by -UseNewEnvironment, only to find it is broken as per this bug.

I guess security might be a consideration though.

PS I'm another Steve Lee :)

[raducoriu]

@SteveL-MSFT Any updates on this issue? I see no changes related to -UseNewEnvironment

[SteveL-MSFT]

I'll bring this up to the WG for discussion. I'm currently thinking maybe a completely new switch instead of reusing -UseNewEnvironment is best to avoid any potential breaks. Maybe call it -UseInitialEnvironment which would be the environment when initial session state was first created. Separately, I think adding -Environment is not controversial and I can work on a PR for that.

[iSazonov]

If we add -Environment we will have Add/Modify/Remove operations and we will need only one option that works as a logon shell (builds the original work environment).

[SteveL-MSFT]

@PowerShell/wg-powershell-cmdlets reviewed this and agreed with the asked behavior and a new switch to avoid any breaking change

[danstur]

Agreed re intent, @SteveL-MSFT, so if we wanted to fix this feature (and also make it work on Unix), we could cache a copy of the environment-variable block on PowerShell startup (before reading profiles), and use that.

Not sure how far along this issue is, but a "common" (well the only one I ever came across) for this feature is to support a scenario where you run a program that changes the "global" environment (i.e. the registry entries) and then run another program that relies on these changes.

The simplest example is a script that runs an installer and then wants to run the installed program right afterwards:

git.exe  /SILENT
# git not in path :(
git clone foo

Which means that the "copy environment at the start of the process would not help, since this is not necessarily the same as the environment at the time the command is executed.

On Windows at least I'd think that using CreateEnvironmentBlock with bInherit set to false would be the correct way to handle this - might even work correctly when specifying Credentials. Would require some parsing of the environment block to use it with the C# Prozess API, but that seems pretty straight-forward.
No solution for linux land though.