Skip to content

Crash with ansible in become from root to user #734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mowgli opened this issue Jul 23, 2020 · 11 comments
Open

Crash with ansible in become from root to user #734

mowgli opened this issue Jul 23, 2020 · 11 comments

Comments

@mowgli
Copy link

mowgli commented Jul 23, 2020

When I run the following task, ansible in pull mode fails badly with the following stackdump when using mitogen.

- name: Create first configuration
  shell: "echo '{{ gitolite_admin_key }}' > '{{ gitolite_path }}/adminkey.pub' && env -u PERL5LIB gitolite setup -pk '{{ gitolite_path }}/adminkey.pub' && rm '{{ gitolite_path }}/adminkey.pub'"
  args:
     creates: "{{ gitolite_path }}/.gitolite/conf/gitolite.conf"
  become: True
  become_method: su
  become_user: "{{ gitolite_user }}"
# gitolite : Create first configuration ***********************************************************************************
  * tschil                     - FAILED!!! ------------------------------------------------------
    Unexpected failure during module execution.
    Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 140, in run
        res = self._execute()
      File "/usr/lib/python2.7/dist-packages/ansible/executor/task_executor.py", line 612, in _execute
        result = self._handler.run(task_vars=variables)
      File "/etc/ansible/plugins/mitogen/ansible_mitogen/mixins.py", line 121, in run
        return super(ActionModuleMixin, self).run(tmp, task_vars)
      File "/usr/lib/python2.7/dist-packages/ansible/plugins/action/shell.py", line 27, in run
        result = command_action.run(task_vars=task_vars)
      File "/etc/ansible/plugins/mitogen/ansible_mitogen/mixins.py", line 121, in run
        return super(ActionModuleMixin, self).run(tmp, task_vars)
      File "/usr/lib/python2.7/dist-packages/ansible/plugins/action/command.py", line 24, in run
        results = merge_hash(results, self._execute_module(task_vars=task_vars, wrap_async=wrap_async))
      File "/etc/ansible/plugins/mitogen/ansible_mitogen/mixins.py", line 364, in _execute_module
        timeout_secs=self.get_task_timeout_secs(),
      File "/etc/ansible/plugins/mitogen/ansible_mitogen/planner.py", line 573, in invoke
        kwargs=planner.get_kwargs(),
      File "/etc/ansible/plugins/mitogen/ansible_mitogen/connection.py", line 451, in call
        return self._rethrow(recv)
      File "/etc/ansible/plugins/mitogen/ansible_mitogen/connection.py", line 437, in _rethrow
        return recv.get().unpickle()
      File "/etc/ansible/plugins/mitogen/mitogen/core.py", line 963, in unpickle
        raise obj
    CallError: exceptions.OSError: [Errno 13] Permission denied: '/root/.ansible/pull/tschil.ethgen.de'
      File "<stdin>", line 3669, in _dispatch_one
      File "master:/etc/ansible/plugins/mitogen/ansible_mitogen/target.py", line 422, in run_module
        return impl.run()
      File "master:/etc/ansible/plugins/mitogen/ansible_mitogen/runner.py", line 440, in run
        self.setup()
      File "master:/etc/ansible/plugins/mitogen/ansible_mitogen/runner.py", line 851, in setup
        super(NewStyleRunner, self).setup()
      File "master:/etc/ansible/plugins/mitogen/ansible_mitogen/runner.py", line 623, in setup
        super(ProgramRunner, self).setup()
      File "master:/etc/ansible/plugins/mitogen/ansible_mitogen/runner.py", line 374, in setup
        self._setup_cwd()
      File "master:/etc/ansible/plugins/mitogen/ansible_mitogen/runner.py", line 384, in _setup_cwd
        os.chdir(self.cwd)
~> ansible --version
ansible 2.7.5
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.13 (default, Sep 26 2018, 18:42:22) [GCC 6.3.0 20170516]
@mowgli
Copy link
Author

mowgli commented Jul 23, 2020

Let me say that the file already exists so the shell command is not meant to run in this play.

And the mitogen version is v0.2.9.

@mowgli
Copy link
Author

mowgli commented Jul 23, 2020

Tested it with the current master but the same result.

@s1113950
Copy link
Collaborator

Can you try these tasks? I'm curious on where it's going to fail. Your error is during Mitogen's setup, where it tries to change into the current working directory. Something about your run can't access the /root/.ansible/pull/tschil.ethgen.de dir, and this should pinpoint it hopefully 🤞 :

---
- name: Create first configuration
  shell: echo "foo"

- name: Create first configuration
  shell: echo "foo"
  args:
     creates: "{{ gitolite_path }}/.gitolite/conf/gitolite.conf"

- name: Create first configuration
  shell: echo "foo"
  become: True

- name: Create first configuration
  shell: echo "foo"
  args:
     creates: "{{ gitolite_path }}/.gitolite/conf/gitolite.conf"
  become: True

- name: Create first configuration
  shell: echo "foo"
  become: True
  become_method: su

- name: Create first configuration
  shell: echo "foo"
  args:
     creates: "{{ gitolite_path }}/.gitolite/conf/gitolite.conf"
  become: True
  become_method: su

- name: Create first configuration
  shell: echo "foo"
  become: True
  become_method: su
  become_user: "{{ gitolite_user }}"

- name: Create first configuration
  shell: echo "foo"
  args:
     creates: "{{ gitolite_path }}/.gitolite/conf/gitolite.conf"
  become: True
  become_method: su
  become_user: "{{ gitolite_user }}"

@mowgli
Copy link
Author

mowgli commented Jul 26, 2020

As expected, the first su to the non-root user fail

The directory /root/.ansible is mode 700. It was automatically created by ansible and this should not be a factor as the chdir should happen before the su happen. For security it makes sense to chdir to / before any rights (de)escalation.

How does ansible itself solves that?

TASK [gitolite : Create first configuration] ****************************************************************************************************************************
changed: [tschil] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": true, "cmd": "echo \"foo\"", "delta": "0:00:00.004260", "end": "2020-07-26 10:09:48.075647", "rc": 0, "start": "2020-07-26 10:09:48.071387", "stderr": "", "stderr_lines": [], "stdout": "foo", "stdout_lines": ["foo"]}

TASK [gitolite : Create first configuration] ****************************************************************************************************************************
ok: [tschil] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": false, "cmd": "echo \"foo\"", "rc": 0, "stdout": "skipped, since /home/gitolite/.gitolite/conf/gitolite.conf exists", "stdout_lines": ["skipped, since /home/gitolite/.gitolite/conf/gitolite.conf exists"]}

TASK [gitolite : Create first configuration] ****************************************************************************************************************************
changed: [tschil] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": true, "cmd": "echo \"foo\"", "delta": "0:00:00.004058", "end": "2020-07-26 10:09:48.343395", "rc": 0, "start": "2020-07-26 10:09:48.339337", "stderr": "", "stderr_lines": [], "stdout": "foo", "stdout_lines": ["foo"]}

TASK [gitolite : Create first configuration] ****************************************************************************************************************************
ok: [tschil] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": false, "cmd": "echo \"foo\"", "rc": 0, "stdout": "skipped, since /home/gitolite/.gitolite/conf/gitolite.conf exists", "stdout_lines": ["skipped, since /home/gitolite/.gitolite/conf/gitolite.conf exists"]}

TASK [gitolite : Create first configuration] ****************************************************************************************************************************
changed: [tschil] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": true, "cmd": "echo \"foo\"", "delta": "0:00:00.005166", "end": "2020-07-26 10:09:48.717165", "rc": 0, "start": "2020-07-26 10:09:48.711999", "stderr": "", "stderr_lines": [], "stdout": "foo", "stdout_lines": ["foo"]}

TASK [gitolite : Create first configuration] ****************************************************************************************************************************
ok: [tschil] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": false, "cmd": "echo \"foo\"", "rc": 0, "stdout": "skipped, since /home/gitolite/.gitolite/conf/gitolite.conf exists", "stdout_lines": ["skipped, since /home/gitolite/.gitolite/conf/gitolite.conf exists"]}

TASK [gitolite : Create first configuration] ****************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was:     os.chdir(self.cwd)
fatal: [tschil]: FAILED! => {"msg": "Unexpected failure during module execution.", "stdout": ""}

@s1113950
Copy link
Collaborator

Ansible makes shell calls to run lots of things, compared to Mitogen which does most of its things with native Python.

You state ...this should not be a factor as the chdir should happen before the su happen; this is what is happening 😅
The chdir can't happen though if permissions don't allow it to. It's similar to this:

$ sudo mkdir ownedByRoot
$ sudo chmod 700 ownedByRoot/
$ cd ownedByRoot/
-bash: cd: ownedByRoot/: Permission denied

Can you run the playbook I posted before without Mitogen? I would think it should still fail because of this block here:

become: True
become_method: su
become_user: "{{ gitolite_user }}"

This block ^^^ is telling Ansible/Mitogen to run a task via su as the {{ gitolite_user }}, but Ansible/Mitogen can't check the creates: part because the {{ gitolite_user }} can't access a root dir that has perms 700.

@mowgli
Copy link
Author

mowgli commented Jul 28, 2020

Sure I can do it. But I don't expect something new. the playbook worked before I migrated to mitogen.

Let me tell you, that I even added a chdir: / to the now broken task but without any success.

When I use ansible-pull, I have no influence, which rights /root/.ansible would have as that is controlled by ansible itself.

So, in between the playbook above run without problem commenting out mitogen from ansible.cfg.

Your use case doesn't meet in that case. Remember, ansible-pull is run by root and does (in this case) a deescalation to a unprivileged user. Further more, I don't see why the check for existence should also be done after the su (even if it should be safe to do so to support the other way around, changing to root).

About your chdir, I think, you miss something. There is no explicit chdir into the directory with no rights. And the gitolite_user has also full rights to gitolite_path. Remember, I speak about the chdir, done before executing (or checking) in the shell module. Eventually explicit given as described above but at least to /.

Let me summary that:

  • The playbook is run by ansible-pull by root (via cron)
  • The directory /root/.ansible is created by ansible with mode 0700.
  • It doesn't matter if I add args: {chdir: /}
  • With plain ansible the playbook run without complaining
  • With mitogen, it error out with the su (I expect there to be some implicite chdir into that directory)
  • We talk about an deescalation, so the transition is from root to gitolite_user, not the way around as it would be when using ansible by ssh to an unprivileged user.

Sorry that I cannot dive into the python code. My python knowledge is very limited. I came from the perl side. ;-)

@s1113950
Copy link
Collaborator

Thanks for this info! It's definitely a bug in Mitogen if it works in Ansible without Mitogen. I think there's enough info here to reproduce the problem on my end; I'll try to get to this issue in a week or two if it hasn't been picked up by someone else in that time (currently banging my head on trying to get #723 to work)

@s1113950
Copy link
Collaborator

s1113950 commented Aug 16, 2020

@mowgli do you have a simple standalone playbook I could use to replicate the issue? I tried replicating the issue but I don't use ansible-pull so am not sure how you ran things.

Here's what I tried running inside a container:
Centos8 OS
Ansible 2.7.5
Python 2
Mitogen master

- name: make dir owned by root
  become: True
  file:
    path: /root/.ansible
    state: directory
    mode: '0600'
  vars:
    ansible_python_interpreter: "/usr/libexec/platform-python"

- name: create new test user in container
  become: True
  user:
    name: dummy
    shell: /bin/bash
  vars:
    ansible_python_interpreter: "/usr/libexec/platform-python"

- name: run something as a different user
  shell: echo "foo"
  become: True
  become_method: su
  become_user: dummy
  vars:
    ansible_python_interpreter: "/usr/libexec/platform-python"

but it worked because the container doesn't have anything in the /root/.ansible dir.

@mowgli
Copy link
Author

mowgli commented Aug 16, 2020 via email

@s1113950
Copy link
Collaborator

s1113950 commented Aug 18, 2020

@mowgli thanks so much for the repro steps! :D
I'll need to put this on hold until Ansible 2.10/collections support is finished: #715 (comment) , but will resume this after that point.

If you'd like to take a look in the meantime, you could play around with where the error happens: https://github.com/dw/mitogen/blob/master/ansible_mitogen/runner.py#L384

Here's how I test Mitogen patches: https://github.com/s1113950/mitogen-test and what I was doing to try and get your issue to work:

TEST=perms CONTAINER_IMAGE=centos:8 ANSIBLE_VERSION=2.7.5 PYTHON_VERSION=2 ANSIBLE_SSH_PASS={mySSHPasswordHereForConvenience} USE_LOCAL_MITOGEN={pathToMitogenRepo} make run-test

and then for debugging, I like the epdb package. Works the same as pdb but with more features: import epdb; epdb.set_trace()

@s1113950 s1113950 reopened this Aug 18, 2020
@s1113950
Copy link
Collaborator

Also sorry for closing, was a misclick when I meant to hit comment 😅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants