When trying to develop and debug in container environment, in lieu of local machine

Contact info

Yueteng | yh958@cornell.edu

Note

Multiple versions of docs are kept in this page in the following order:

• Newer versions towards the top
• Older  versions towards the end

Doc Version 1.1

Purposes

Solved these problems:

1. Two containers (one for VS Code, the other one for running) appear in Docker Desktop
   1. → Would like to merge them into one
   2. → Even better, remove the container after finishing using (clean for the eye)
2. After making changes to the code, the changes are not auto-reloaded. Must re-run docker-compose up
   1. → Enable auto-reloading
3. Env variables not all enabled
   1. → Re-enable all previously used development env variables, with auto-reloading enabled (via volume)
   2. → Create a separate prod env variables, with auto-reloading disabled (via volume)
4. VS Code extension Python does not (always) work
   1. → Fix the "couldn't load python tool" error

Step-by-step guide

Step 0 Preparation; Overview

docker volume rm $(docker volume ls -f dangling=true -q)

In terminal, run the above command to do some cleaning. (There might be some dangling volumes not garbage-collected yet by docker).

(Optional) In Docker Desktop, on the Container / Apps tab, delete all related containers of this project.

(Optional) In Docker Desktop, on the Images tab, except busybox , delete all related images of this project. Do NOT delete busybox.

Switch to git branch  docker2 , where the new code resides.

If everything goes swimmingly, it is recommended to merge docker2 into the branch you're working on (master, feature, etc.).

Some overview:

• Python image issues:
  • The current Python image used in this project is based on alpine Linux.
  • alpine Linux does not include many necessary libs for development, explaining why Python extension fails and additional lib installation (RUN apk *) in Dockerfile
  • We're going to switch to the standard Python image to avoid all the problems.
• VS Code and container
  • Previously, we let VS Code create a container and develop inside (remember the container name of random scientist/mathematician), hence the extra container.
  • Now, we're going to build/create/run the container by ourselves, and let VS Code attach to the container. Only ONE both for running and VS Code
  • Additionally, upon finished (developing and running), we'll shut down the container, and it will be removed from Docker Desktop > Containers Apps, making it look clean.
• Others (if you're interested, optional)
  • Dockerfile: The base image FROM is changed to python:3.8.6, a full-feature image. You will also see many lines commented out, unnecessary for the moment.
  • docker-compose.yml : This is used for development (also the default one). You can see all env variables for  development.
  • docker-compose-prod.yml: This will be for production. Not finished yet.
  • docker-compose.debug.yml.old: Just some backup. Ignore it.

Step 1 Run the container

# Path for online repo
backend-online/dev-backend
 
# Path for offline repo
backend-offline/dev-dashboard

Go to the path as above.

# For the first time
# Or whenever you want to update the image, e.g changing Dockerfile, requirements.txt
docker-compose up -d --build
 
# Otherwise
docker-compose up -d

Run the command.

This will build the image, create+run the container.

This could take a minute.

Note: If there is an error message says "cannot import name 'Markup' from 'jinja2', you can try to fix it through the means provided by ImportError: cannot import name 'Markup' from 'jinja2'. (Basically type python3 in terminal, then type  'from jinja2.utils import markupsafe', then  'markupsafe.Markup()'. The error will likely to be fixed.

If you are using Mac and get an error message when running the container for the first time, saying "Ports are not available: listen tcp 0.0.0.0:5000: bind: address already in use", it is likely that your Mac AirPlay server is running on this port (even when you try to kill -9 the process, it restarts automatically). To solve this problem, deactivate the AirPlay Receiver by:

Go to System Preferences › Sharing, and unchecking AirPlay Receiver to release port 5000

You should see a new container running.

Name is based on the path (not random anymore)

On VS Code, make sure you've installed these two extensions:

• Remote - Containers
• Python

Step 2 Attach VS Code to the container; Necessary extensions

On VS Code, click the Remote button at bottom left > Attach to running container > Click the container name in the menu.

For the first time, it could take a minute to load.

For the first time, click File > Open and enter the path above /var/www/ . Then you will be at the working directory.

From the second time, VS Code will remember to open this path automatically.

As always, make sure you've installed this Python extension, and it is enabled in container.

Click an .py file to activate it.

Choose interpreter of Python 3.8.6 64-bit

It might show loading/downloading analysis at the bottom status bar, after which it should work properly.

Step 3 Run and debug; Auto-reload

Everything under www, if changed:

• Will be auto-reloaded
• Will will make change in your repo's docker_code subfolder
• You can think of www and docker_code as two pointers to the same folder

# Visit http://localhost:5000/
{
  "message": "Welcome to the Dockerized Flask MongoDB app, again and again and again (this will auto-reload)!"
}
 
# Change the message at app/views/static.py
# For instance, add "test" at the end
 
# Visit http://localhost:5000/ again
# Notice the end now includes "test"
# It's auto-reloaded
{
  "message": "Welcome to the Dockerized Flask MongoDB app, again and again and again (this will auto-reload)! test"
}

This example is only for online repo. (Offline repo does not have defined this url in repo)

Try the above to see auto-reloading is working.

Step 4 Close connexion with container; Shut down container

On VS Code, click the Remote button at bottom left > Close remote connection.

docker-compose down

In terminal, run the command above. It will shutdown and remove container (as you can see it disappears from Docker Desktop > Containers / Apps)

Doc Version 1.0

Purpose

1. Environment consistency: Avoiding inconsistency due to different dependency/lib on local machine v in container
2. Centralised dependency/lib management: Only one member needs to update the config (requirements.txt) in docker, and all other members will have access to updated dependency/libs automatically after git pull
3. Convenience: Avoiding install/keep the same dependency/lib twice, i.e. on local machine and in container

Step-by-step guide

Note: This guide is based on VS Code. I've also tried IntelliJ IDEA (including PyCharm), but unfortunately it doesn't work on my laptop. If interested, see the link below under "Read more"

Step 1 Install extension and launch docker

On VS Code, make sure you've installed these two extensions:

• Remote - Containers
• Python
  

After installing, you should see a button for this extension at the bottom-left corner of VS Code

Launch docker (you don't need to start the project container)

Step 2 Open container

Click the remote container button > Open folder in container > Open your working directory with Dockerfile inside the repo. For instance, for backend-offline subteam, choose backend-offline/dev-dashboard/dashboard-backend

You should now see a progress notification at the bottom-right corner. Note that this will take a minute (Only for the first time opening the working directory. Next time it would start very quickly).

When completed, you'll notice VS Code has deployed an image (name starting with vsc-) in your docker, and has inserted/started an container (name including some random scientist/mathematician).

Step 3 Run and debug in container environment

Click the button to the right of the container button, to choose your interpreter/SDK.

Note: If you don't see the button:

• Go to Extension > Python, and make sure it is "enabled in dev container"
• Double-click a .py file to activate Python extension.

For python, click the area, which shows Python 3.6.8 64-bit above, and choose the interpreter Python 3.6.8 64-bit (it should be at /usr/local/bin/python)

Now you're ready to develop, run, and debug in container!

To exit, simple click the remote container button at the bottom-left corner of VS Code > Close Remote Connection.

{'_id': ObjectId('5f8771822ca2e28e9b5572df'), 'test_field': 'test_value'}
Linux-4.19.76-linuxkit-x86_64-with#1 SMP Tue May 26 11:42:35 UTC 2020

I've included a simple code in your repo:

• backend-offline subteam: in master branch, at backend-offline/dev-dashboard/dashboard-backend/temp_container_test_run/container_test_run.py
• backend-online subteam: in master branch, at backend-online/dev-backend/flask_docker/temp_container_test_run/container_test_run.py

If you run it, it should show in terminal the same output as above. Note the 2nd line Linux shows it's been run in container (not local machine of Mac OS/Windows)

Note

Everything you've coded is done in the same git repo.

If you choose to stop the container after finish coding, it's alright. Next time VS Code will automatically start it when you open the working directory as in steps above

Read more

Guide from VS code: https://code.visualstudio.com/docs/remote/create-dev-container

Guide from JetBrains (IntelliJ IDEA, PyCharm): https://www.jetbrains.com/help/idea/configuring-remote-python-sdks.html#Docker