`module show` in modules relying on variables set in other modules fails

[Flamefire]

Describe the bug

We have modules that reference variables of other modules like:
setenv("CUOBJDUMP_PATH", os.getenv("CUDA_HOME") .. "/bin/cubjdump")
as CUDA_HOME is not set in a clean environment (no modules loaded) this fails with "attempt to concatenate a nil value"

To Reproduce

Have a module with e.g.:

depends_on("CUDA/12.6.0")
setenv("CUOBJDUMP_PATH", os.getenv("CUDA_HOME") .. "/bin/cubjdump")

Call module show <thatmodule> and observe the bug while module load <thatmodule> works.

Expected behavior

I'm not fully sure what a solution would be but would like to ask for ideas as the issue seems to be easy to encounter.
A downstream fix might be something like os.getenv("CUDA_HOME") or "$CUDA_HOME" but that has 2 problems:
When the variable is unset or empty on module load it will hide that silently causing later subtle issues.
So maybe LMod has better means of handling this only during module show, maybe something like getenv_with_default_on_module_show

[rtmclay]

Lmod can use a Lua trick to change the definition of os.getenv() when in show mode so that it returns <unknown>. Will that work for you?

Using this style of module coding has a problem on unload. On unload CUDA_HOME won't have a value. It won't matter when you use setenv() because Lmod doesn't care what the value is. However, if you did a prepend_path like this:

depends_on("CUDA/12.6.0")
local foo_path = pathJoin(os.getenv("CUDA_HOME") or "", "/bin/foo")
prepend_path("PATH", foo_path)

the variable foo_path won't have the same value because CUDA_HOME is unset and therefore the original path entry will not be removed. Depending on environment variables outside of where there are defined is dangerous and should be avoided.

[Flamefire]

What is the alternative when you have to refer to a variable set in another module, or at least to the software root of another module?
The example given above is actually quite common: Some software needs a custom variable set to the path to a binary when that isn't in a standard location like /usr. So I'm afraid this comes up a couple times at least

And why won't CUDA_HOME have a value on unload? Shouldn't it be unloaded before CUDA so the variable is still set?

[rtmclay]

If this is something that you do regularly, you can provide your own function in your SitePackage.lua or /etc/lmod/lmod_config.lua which might be something like:

function protected_getenv(name,defaultValue)
    local result = os.getenv(name)
    if (result) then
       return result
    end
    return (mode() == "show") and defaultValue or nil
end

sandbox_registration { protected_getenv = protected_getenv }

Then you can use it like this:

depends_on("CUDA/12.6.0")
setenv("CUOBJDUMP_PATH", pathJoin(protected_getenv("CUDA_HOME","$CUDA_HOME") ,"/bin/cubjdump"))

You might want to use pathJoin() as it correctly gets the slashes right.

[rtmclay]

As to my comments about this style of coding where you define environment variables in one module being dangeous, I have thought about it some more. I have changed the way Lmod works. Now Lmod only unsets environment variables in the user's environment, not the Lmod process. In this way CUDA_HOME would still have a value after CUDA/12.6.0 has been unloaded.

This will be live when Lmod 9.2.6+ is released.

[Flamefire]

Ok, great!
We do this regularly (now) in EasyBuild. So we would need to define such a function in all affected modules as we cannot require users (HPC admins) to modify their local configs.
So it would be great if Lmod provided such a function. I guess in "show" mode it could always return $FOO if the variable is not set. Or at least as a default, so one can in most cases do showsave_getenv("FOO") (just a name I came up now, there might be something better, but if it is in LMod I'd guess indicating the handling of show-mode makes sense)

[rtmclay]

Branch IS840-getenv now has a new function that has defaults:

protected_getenv{"FOO","$FOO",modeA={"show",...}}
protected_getenv{"FOO","$FOO"}  -- modeA={"show"} assumed
protected_getenv{"FOO"}                -- "$FOO" and modeA={"show"} assumed

So this function is just a wrapper for os.getenv() except when the mode() function matches a name in modeA and os.getenv() returns nil. In that case it returns the string in the second argument.

Please test this branch and report back how it works for you.

[rtmclay]

I do have a question on how Lmod is installed on sites that use EasyBuild. In particular, how do you set any Lmod environment variables that are not the default. Thanks!

[Flamefire]

how do you set any Lmod environment variables that are not the default. Thanks!

We have an (/etc/)init script that sources the lmod init script after setting the custom config variables including LMOD_SYSTEM_NAME based on /proc/cpuinfo so we can use the same script on all clusters