Failure during tofu call : fork/exec /usr/bin/tenv: inappropriate ioctl for device

[mykyta-martsevyi]

Describe the bug
After new version (v4.0.0) released my github actions have started failing during installation and running tofu

To Reproduce
Steps to reproduce the behavior:
Just try to install tenv v4.0.0 and run tenv tofu install && tofu version
Reverted to v3.2.11 and it works again

Expected behavior
Expect to install and run tofu without any errors

Screenshots

Environment (please complete the following information):

• OS: ubuntu-22.04
• tenv version: 4.0.1

Additional context
Add any other context about the problem here.

[dyegoe]

Same error, but in another setup. I am getting it from pre-commit.

OS: Fedora 41
tenv version: 4.0.1

> pre-commit run
OpenTofu fmt.............................................................Failed
- hook id: tofu_fmt
- exit code: 1

Failure during tofu call : fork/exec /home/dyego/.local/bin/tenv: inappropriate ioctl for device

OpenTofu validate........................................................Failed
- hook id: tofu_validate
- exit code: 1

Validation failed: .
Failure during tofu call : fork/exec /home/dyego/.local/bin/tenv: inappropriate ioctl for device

OpenTofu docs............................................................Passed
OpenTofu validate with tflint............................................Passed
check yaml...............................................................Passed
detect aws credentials...................................................Passed
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed

[dyegoe]

The breaking change was introduced v4.0.0 where the default value for TENV_DETACHED_PROXY is now true as we can see it here.
L35 of the versionmanager/proxy/detach/detached_others.go

If you export it as false, everything works

> export TENV_DETACHED_PROXY=false
> pre-commit run -a
OpenTofu fmt.............................................................Passed
OpenTofu validate........................................................Passed
OpenTofu docs............................................................Passed
OpenTofu validate with tflint............................................Passed
check yaml...............................................................Passed
detect aws credentials...................................................Passed
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed

The question is: is it the expected behaviour? Should we start to export it as false?

[iam-take]

This behavior change was a feature of version 4.0.0 as a result of #255. I do wonder if this has been checked for the proper impact as all of our pipelines on Azure DevOps are now also having the same issue. Our agents run on Ubuntu 22:04

[dvaumoron]

Sorry, we will need a better handling to allow tenv to detect properly if the detached proxy behavior is needed...

[tspearconquest]

It can be done by simply checking for environment variable CI having the true value as I mentioned in the other issue. Most CI environments set this variable by default. If someone on each of: Bitbucket, GitHub and Azure DevOps could confirm, that would be appreciated. I can confirm for Jenkins and Gitlab at least.

[tspearconquest]

Also can confirm that setting TENV_DETACHED_PROXY to false fixes my pipelines.

[tspearconquest]

This behavior change was a feature of version 4.0.0 as a result of #255. I do wonder if this has been checked for the proper impact as all of our pipelines on Azure DevOps are now also having the same issue. Our agents run on Ubuntu 22:04

The fact that it was a major version release is indicative of potentially breaking changes.

[tspearconquest]

#308 should hopefully fix this -- assuming that I'm correct in that CI=true is used across all of the big CI players.

[iam-take]

This behavior change was a feature of version 4.0.0 as a result of #255. I do wonder if this has been checked for the proper impact as all of our pipelines on Azure DevOps are now also having the same issue. Our agents run on Ubuntu 22:04

The fact that it was a major version release is indicative of potentially breaking changes.

Whilst I do understand that a Major release can contain breaking changes @tspearconquest the release notes do not seem to reflect or indicate these changes are breaking. I noticed a while back this was the same for another release as well. It would be greatly appriciated if the release notes give a upgrade notice of these breaking changes. As currently we need to dig through the issue's / pr's to see what has been changed.

It can be done by simply checking for environment variable CI having the true value as I mentioned in the other issue. Most CI environments set this variable by default. If someone on GitHub and Azure DevOps could confirm, that would be appreciated. I can confirm for Bitbucket and Gitlab at least.

For Azure DevOps I can confirm that this is not a default variable being set.

[tspearconquest]

Thanks for confirming. I've opened a PR to add a check for the CI variable. Could you let me know if Azure DevOps has any variable set by default that could be checked?

[tspearconquest]

Did some investigation of my own in the mean time - https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml#agent-variables-devops-services

I found 2 AZURE_ variables; one of which I believe is used also by the Azure CLI and so could potentially be set in non-CI environments, and the other which I'm not certain is always set. However, I also found PIPELINE_WORKSPACE which is always set and is certain to be only set in a pipeline, so I've now added that to my patch alongside the CI variable.

[kvendingoldo]

It would be greatly appriciated if the release notes give a upgrade notice of these breaking changes

I totally agree with you. We'll be working on that at the 1st priority with @Nmishin in scope of #17

[ThibaultNocchi]

#308 should hopefully fix this -- assuming that I'm correct in that CI=true is used across all of the big CI players.

I confirm that Gitlab CI has the $CI variable set to true.

[dvaumoron]

accepted #308, improved it with #309, and created a new release (v4.1.0)

for further discussion on breaking change management, please go to #310

[tspearconquest]

Hi, unfortunately this fix wasn't sufficient for Azure DevOps after all. I'm getting this error when setting PIPELINE_WORKSPACE to a path (as Azure DevOps does, and I failed to notice/test, instead assuming a boolean check of the variable would be sufficient):

Failed to read CI or PIPELINE_WORKSPACE environment variable, keep detached default behavior to true : strconv.ParseBool: parsing "/tmp": invalid syntax

I set it in a manual run as below to produce the above:

cd /tmp; PIPELINE_WORKSPACE=/tmp tenv tofu install

Please accept my apologies.

[tspearconquest]

Please reopen while I work on a better fix

[tspearconquest]

Hi @kvendingoldo I'm having some trouble; Go is not my specialty and I'm pretty novice at it. I think the issue is caused from getenv.BoolFallback assuming all environment variables are boolean so I wanted to add a check in config/utils/utils.go to handle this string environment variable. I think we can simply try to determine if it is set to a non-empty string, and if so, have BoolFallback return true. Is that the right route to take, and if so, is it acceptable to add an import for reflect in the same to allow for type checking?

[ego93]

I'm having a similar issue to this main issue, getting Failure during terraform call : fork/exec /opt/homebrew/bin/tenv: operation not supported by device with pre-commit using antonbabenko/pre-commit-terraform 1.96.3 terraform_validate hook

I'm on tenv 4.1.0

[kvendingoldo]

@tspearconquest it sounds reasonable for me, and I do this in some of my projects. @dvaumoron how it looks for you?

[dvaumoron]

I think for such case we can have an other method to handle env var presence without assuming a boolean value, i can do that change later today

[tspearconquest]

Great, thanks. Meantime I will look into the pre-commit environment for a variable to check, although it sounds like we're getting "into the weeds" with this mechanism and it might be better to rethink how to detect if there is a TTY and whether to use that or detach; which is the root cause of the original change and the current ongoing reports.

…

On Fri, Dec 27, 2024, 8:12 AM Denis Vaumoron ***@***.***> wrote: I think for such case we can have an other method to handle env var presence without assuming a boolean value, i can do that change later today — Reply to this email directly, view it on GitHub <#305 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ATRTFZ2SMUSAK263ZYHTDET2HVN5TAVCNFSM6AAAAABUCZQUWSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNRTG42DAMJZGY> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[tspearconquest]

https://rderik.com/blog/identify-if-output-goes-to-the-terminal-or-is-being-redirected-in-golang/ this looks promising for possibly detecting a TTY. WDYT?

Found this while googling for a way to detect the presence of a TTY.

[dvaumoron]

Great, thanks, I will use that on top level proxy (tenv binary as intermediate proxy has already it's input and output redirected)

[dvaumoron]

@tspearconquest , @ego93 , @ThibaultNocchi , @dyegoe , @iam-take , @mykyta-martsevyi

can you test https://github.com/tofuutils/tenv/releases/tag/v4.2.0-beta1 ? With information on which environement

[ThibaultNocchi]

@tspearconquest , @ego93 , @ThibaultNocchi , @dyegoe , @iam-take , @mykyta-martsevyi

can you test https://github.com/tofuutils/tenv/releases/tag/v4.2.0-beta1 ? With information on which environement

I'm away for the holidays, I'll be back to work on Thursday :)

[iam-take]

@dvaumoron I have tested it on Azure DevOps with the default host pool given by Microsoft. and this works. Also it works for our self-hosted docker containers. (Ubuntu 22:04)

Tested on:
Azure DevOps
Ubuntu based agent (Microsoft Hosted)

Below a very quick example Azure DevOps pipeline to test this:

trigger:
- main

pool:
  vmImage: ubuntu-latest

steps:
- script: |
    export TENV_AUTO_INSTALL=true
    Version="v4.2.0-beta1"
    echo "Installing Tenv with version $Version"
    curl -O -L "https://github.com/tofuutils/tenv/releases/download/$Version/tenv_${Version}_amd64.deb"
    sudo dpkg -i "tenv_${Version}_amd64.deb"
    tenv --version
    echo "Installing example version of Terraform"
    tenv tf use 1.9.8
   terraform init
  displayName: 'TENV Test'

[dyegoe]

@dvaumoron
I downloaded the v4.2.0-beta1, and it still doesn't work for pre-commit

> env | grep -i tenv
TENV_AUTO_INSTALL=true
TENV_ROOT=/home/dyego/.config/tenv
> tenv version
tenv version v4.2.0-beta1
> pre-commit run -a
OpenTofu fmt.............................................................Failed
- hook id: tofu_fmt
- exit code: 1

Failure during tofu call : fork/exec /home/dyego/.local/bin/tenv: inappropriate ioctl for device
Failure during tofu call : fork/exec /home/dyego/.local/bin/tenv: inappropriate ioctl for device
Failure during tofu call : fork/exec /home/dyego/.local/bin/tenv: inappropriate ioctl for device

OpenTofu validate........................................................Passed
OpenTofu docs............................................................Passed
OpenTofu validate with tflint............................................Passed
check yaml...............................................................Passed
detect aws credentials...................................................Passed
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed

If export TENV_DETACHED_PROXY=false, then everything looks fine as the previous version 4.0.1.
Let me know if you need further tests.

[dvaumoron]

@dyegoe , i don't understand yet why it does not work with pre-commit (it seems the proxy react as if it has access to a terminal even if it did not...)

[ego93]

@dvaumoron
I've tested pre-commit and I'm still getting an error see below:

> tenv version
tenv version v4.2.0-beta1
> pre-commit run --all-files
Terraform fmt............................................................Failed
- hook id: terraform_fmt
- exit code: 1

Failure during terraform call : fork/exec /usr/local/bin/tenv: operation not supported by device

Terraform docs...........................................................Passed
Terraform validate with tflint...........................................Passed
Terraform validate.......................................................Passed
check for merge conflicts................................................Passed
fix end of files.........................................................Passed
trim trailing whitespace.................................................Passed
mixed line ending........................................................Passed

pre-commit 4.0.1
antonbabenko/pre-commit-terraform v1.96.3 hook
MacOS 15.2

Like with @dyegoe result if I set export TENV_DETACHED_PROXY=false it works fine

[tspearconquest]

Tested v4.2.0-beta1 and its working fine for Jenkins. Hope the issue with pre-commit can be isolated soon and fixed to get this out to everyone. Thanks for the efforts on this!

[kvendingoldo]

moved issue with pre-commit to a separate ticket: #328

[ThibaultNocchi]

@tspearconquest , @ego93 , @ThibaultNocchi , @dyegoe , @iam-take , @mykyta-martsevyi

can you test v4.2.0-beta1 (release) ? With information on which environement

It's all good for me in Gitlab CI without any added environment variable.

[archoversight]

Question:

Why is the original terraform proxy and tenv proxy sticking around as a process in the first place? Why not call execvp (C stdlib) or whatever the equivalent is to replace the original tenv binary with the resulting binary?

When I run:

terraform fmt -

I am left with three processes running:

-+= 22616 archoversight terraform fmt -
 \-+= 22617 archoversight tenv call terraform fmt -
   \--= 22618 archoversight /Users/archoversight/.tenv/Terraform/1.5.7/terraform fmt -

This complexity of trying to "proxy" stdout/stdin and the signals is unnecessary on macOS/Linux/others where replacing the process wholesale will allow it to correctly capture signals and doesn't require anything special.

[dvaumoron]

I used the go standard library os/exec package, moreover execvp is not a cross platform syscall.

[archoversight]

I used the go standard library os/exec package, moreover execvp is not a cross platform syscall.

Right, and that package creates a sub-process and sets up pipes for stdout/stdin, which is what the issue is.

Instead you would want to use:

https://pkg.go.dev/syscall#Exec

On Linux/macOS/FreeBSD/others and do something special on Windows only...

This way you don't create a separate sub-process that is part of the process group, and don't need to do all the extra work of proxying stdin/stdout and it just replaces the process.