-
Notifications
You must be signed in to change notification settings - Fork 159
RFC: Use RunTask API for attached runs
This is a proposal for moving attached runs (emp run without the -d flag) to use the ECS RunTask API, instead of using the Docker Daemon API directly. The benefits of this are that:
- It means containers started via an attached run can use ECS Task Roles.
- It normalizes how attached/detached containers are ran.
- Paves the way for the possibility of adding an
emp attachcommand.
Currently, "attached runs" are NOT implemented in the CloudFormation Scheduler, but rather in the Docker Scheduler.
When you run a command like emp run bash, Empire uses the Docker scheduler, connects to the specified Docker daemon, and uses the CreateContainer, StartContainer, and AttachToContainer API's.
I'm proposing that we instead just use the RunTask API (which detached runs already use), and use the Docker Daemon AttachToContainer API to attach to the container on the given host, after ECS has started it.
When a container is started via ECS, Empire will need to connect to the Docker Daemon on the host that ECS started the container on. Empire will open a new connection for each attach run.
Since ECS doesn't expose any information about the Docker container ID. Empire will use ListContainers to find containers where the com.amazonaws.ecs.task-arn label matches the started task, then attach to this container. I believe that the RunTask API will return before the actual Docker container is running, so Empire will need to wait for the task to enter the RUNNING state before connecting.
At the moment, attached runs are ran by the Docker daemon specified at start up (--docker.socket/DOCKER_HOST). This is also the daemon that Empire uses to pull images, when extracting a Procfile.
A new flag would be added to allow you to specify some container instance constraints for where attached runs would be ran, so that you can limit attached runs to a specific host or set of hosts.
After this change, the DOCKER_HOST flag would only be used to tell Empire what docker daemon to use.
Empire should assume that that Docker Daemon is exposed over TLS when connecting to the container instance.
Race condition between starting the task and attaching to the container
Since RunTask will return before the Docker container is started, Empire will need to poll the tasks status, and connect when it enters the 'RUNNING' state. It's possible that between this state change, and the time that Empire attaches to the container, the container could have stopped.
This is also true of the existing implementation.