Branch: refs/heads/master Author: Enrico Tröger enrico.troeger@uvena.de Committer: Enrico Tröger enrico.troeger@uvena.de Date: Sun, 14 Jul 2019 22:19:19 UTC Commit: 1e16f26d23ca3a5599a62ebd2f3d1a99f7980b65 https://github.com/geany/www.geany.org/commit/1e16f26d23ca3a5599a62ebd2f3d1a...
Log Message: ----------- Add Dockerfile and documentation for local development
Alternatively to setting up a virtualenv and a local database, we provide a Dockerfile to create a container image to be ran locally for easier setup.
Quick usage: make docker-build && make docker-run
Modified Paths: -------------- .dockerignore .gitignore Makefile README.dev.md docker/Dockerfile docker/entrypoint.sh docker/irc_userlist docker/local_settings.docker.py geany/settings.py tox.ini
Modified: .dockerignore 43 lines changed, 43 insertions(+), 0 deletions(-) =================================================================== @@ -0,0 +1,43 @@ +**/*.pyc +**/__pycache__ +*.egg +*.egg-info +*.pyc +*.pyd +*.pyo +*.swp +.Python +.cache +.coverage +.coverage +.coverage.* +.dockerignore +.eggs +.git +.gitignore +.tox +.venv +build +dist +docker/data +docs/_build +env +venv + + +# MacOS / Windows stuff +.DocumentRevisions-V100 +.fseventsd +.Spotlight-V100 +.TemporaryItems +.Trashes +.VolumeIcon.icns +.com.apple.timemachine.donotpresent +.DS_Store +.AppleDouble +.LSOverride +.AppleDB +.AppleDesktop +Network Trash Folder +Temporary Items +.apdisk
Modified: .gitignore 1 lines changed, 1 insertions(+), 0 deletions(-) =================================================================== @@ -4,3 +4,4 @@ local_settings.py venv geany/media +docker/data
Modified: Makefile 38 lines changed, 38 insertions(+), 0 deletions(-) =================================================================== @@ -0,0 +1,38 @@ +# -*- coding: utf-8 -*- +# +# LICENCE: This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see http://www.gnu.org/licenses/. +# + + +docker-run: + docker run \ + --rm \ + --interactive=true \ + --tty=true \ + --user "$$(id --user):$$(id --group)" \ + --mount "type=bind,src=$$(pwd),dst=/app" \ + --publish 8000:8000 \ + --name geany_dev \ + geany_dev:latest + + +docker-build: + docker build \ + --file docker/Dockerfile \ + --tag geany_dev \ + . + + +docker-clean: + rm -rf docker/data
Modified: README.dev.md 87 lines changed, 70 insertions(+), 17 deletions(-) =================================================================== @@ -15,8 +15,18 @@ extra packages installed on your system. The code requires Python 3.5 or higher.
-Prepare your system -------------------- +Get the code +------------ + +Perform a usual clone of the www.geany.org repository from Github: + + git clone git://github.com/geany/www.geany.org + + +Local setup using virtualenv +---------------------------- + +### Prepare your system
To ease package handling and to get a clean environment, we use virtualenv. Into the virtualenv we will install Django, Mezzanine and a couple of @@ -31,16 +41,7 @@ packages: apt-get install python3-venv python3-pip python3-dev build-essential libmysqlclient-dev libmemcached-dev
-Get the code ------------- - -Perform a usual clone of the www.geany.org repository from Github: - - git clone git://github.com/geany/www.geany.org - - -Setting up a virtualenv ------------------------ +### Setting up a virtualenv
Change into the freshly cloned repository directory and execute the following commands to create a new virtualenv and install required Python packages: @@ -55,8 +56,7 @@ This will setup a new virtualenv, upgrade the Python package manager pip and install required packages for the website.
-Create a local config ---------------------- +### Create a local config
Use a text editor of choice (we all know what this would be...) and create a new file *local_settings.py* in *www.geany.org/geany/* (next to the existing *settings.py*). @@ -132,8 +132,7 @@ The database dump contains a default admin user: password: change-me
-Start the development server ----------------------------- +### Start the development server
After you set up everything as described above, you are ready to start the development server to actually do something: @@ -152,8 +151,62 @@ to reload the changed file(s). This is very helpful. To stop the server, simply interrupt it with *Ctrl-C*.
+Local setup using Docker +------------------------ + +Alternatively, a Dockerfile is provided to build a Docker image +and to run the website in a Docker container. +This is the easiest way to get a local environment running. + +### Local config + +When using the Docker image, a prepared local settings file is +used with already adjusted settings for running in a Docker container. +This file is located at `docker/local_settings.docker.py`. + + +### Build container image + +First, you need to build the image: + + make docker-build + +This will take some time but is only necessary once. + +Note: before building the image, carefully review the Dockerfile +and especially make sure you use a base image you can trust. + + +### Start the container + +After the image is built, you can start the container: + + make docker-run + +On the first run, the database is setup, screenshots are downloaded to be +locally available as well as a few more preparations. + +Once running, you can open the resulting site in your browser by pointing it +to http://localhost:8000. + +Basically now you are done and you can start improving the website. +A little detail you might notice: once you change any .py file +which is knwon by Django, the development server will restart automatically +to reload the changed file(s). This is very helpful. + +To stop the container, simply interrupt it with *Ctrl-C*. + +### Cleanup + +All files created on first container startup are stored in `docker/data`. +To start from scratch (e.g. with a fresh database, no uploads, etc.), simply +delete this directory or run: + + make docker-clean + + Management Commands ------------------------ +-------------------
In addition to the usual Django management commands (for a list run `python manage.py`), the Geany apps provide a few more.
Modified: docker/Dockerfile 101 lines changed, 101 insertions(+), 0 deletions(-) =================================================================== @@ -0,0 +1,101 @@ +# LICENCE: This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see http://www.gnu.org/licenses/. +# +# +# Docker container to run the Geany website locally for development. +# This is not intended for production!! +# Before using this image, please get yourself familiar with Docker and +# carefully review this Dockerfile before using it! +# +# Build image: +# docker build -f docker/Dockerfile -t geany_dev . +# Run container: +# docker run --rm -it --user "$(id --user):$(id --group)" --mount "type=bind,src=$(pwd),dst=/app" -p 8000:8000 --name geany_dev geany_dev:latest + +# Intermediate build container +FROM debian:stretch-slim AS builder + +ENV LANG=C.UTF-8 \ + DEBIAN_FRONTEND=noninteractive \ + # Extra python env + PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + PIP_DISABLE_PIP_VERSION_CHECK=1 \ + PIP_FORMAT="columns" \ + PIP_NO_BINARY=":all:" \ + PIP_CACHE_DIR="/tmp/pip" \ + PIP_TIMEOUT=60 + +RUN apt-get update && \ + apt-get install --assume-yes --no-install-recommends \ + python3 \ + build-essential \ + git \ + python3-pip \ + python3-wheel \ + python3-venv \ + python3-dev \ + default-libmysqlclient-dev \ + libffi-dev \ + libjpeg-dev \ + libmemcached-dev \ + zlib1g-dev \ + libssl-dev && \ + mkdir -p /app /data /venv && \ + chown -R nobody:nogroup /app /data /venv + +WORKDIR /app +USER nobody:nogroup +COPY requirements.txt /data/ +# Install Python deps +RUN python3 -m venv --copies /venv && \ + /venv/bin/pip install -U pip setuptools && \ + /venv/bin/pip install -r /data/requirements.txt + + +# App container +FROM debian:stretch-slim AS app + +ENV LANG=C.UTF-8 \ + DEBIAN_FRONTEND=noninteractive \ + # Extra python env + PYTHONDONTWRITEBYTECODE=1 \ + PYTHONUNBUFFERED=1 \ + # Path to local settings + LOCAL_SETTINGS_PY=/app/docker/local_settings.docker.py + +# do all of this in one RUN to limit final image size +RUN apt-get update && \ + apt-get install --assume-yes --no-install-recommends \ + ca-certificates \ + intltool \ + libjpeg62-turbo \ + libmariadbclient18 \ + libmemcached11 \ + libssl1.1 \ + python3 \ + sqlite3 \ + wget \ + zlib1g && \ + rm -rf /var/lib/apt/lists/* + +WORKDIR /app +USER nobody:nogroup + +# copy in Python environment +COPY --from=builder /venv /venv +# copy helpers and configs +COPY ./docker/entrypoint.sh ./docker/irc_userlist /data/ + +EXPOSE 8000 +CMD ["/bin/bash", "/data/entrypoint.sh"]
Modified: docker/entrypoint.sh 70 lines changed, 70 insertions(+), 0 deletions(-) =================================================================== @@ -0,0 +1,70 @@ +#!/bin/bash +# +# LICENCE: This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see http://www.gnu.org/licenses/. +# + +set -e + +mkdir -p /app/docker/data + +# setup database if it does not exist +if [ ! -f "/app/docker/data/geany_dev.db" ]; then + echo "==== Setup database ====" + /venv/bin/python manage.py reset_db --noinput + /venv/bin/python manage.py migrate --run-syncdb --noinput + echo 'DELETE FROM auth_permission; + DELETE FROM django_content_type; + DELETE FROM django_site;' | /venv/bin/python manage.py dbshell + /venv/bin/python manage.py loaddata database.json +fi +if [ ! -f "/app/docker/data/geany_dev_nightlybuilds.db" ]; then + /venv/bin/python manage.py migrate --database nightlybuilds --run-syncdb --noinput + /venv/bin/python manage.py loaddata --database nightlybuilds database_nightlybuilds.json +fi + +# screenshots +echo "==== Download screenshots ====" +SCREENSHOTS="$(sqlite3 docker/data/geany_dev.db 'select file from galleries_galleryimage;')" +# add homepage screenshots +SCREENSHOTS="${SCREENSHOTS} +uploads/screenshots/homepage/geany_dark_2019-05-20.png +uploads/screenshots/homepage/geany_light_php-2019-06-15.png +uploads/screenshots/homepage/geany_windows_classic-2019-06-09.png +" +pushd /app/docker/data +for screenshot in ${SCREENSHOTS}; do + if [ ! -f "/app/docker/data/${screenshot}" ]; then + mkdir -p $(dirname "/app/docker/data/${screenshot}") + wget \ + --no-verbose \ + --output-document="/app/docker/data/${screenshot}" \ + "https://www.geany.org/media/$%7Bscreenshot%7D" + fi +done +popd + +# generate i18n stats if not already available +if [ ! -d "/app/docker/data/i18n" ]; then + echo "==== Update I18N statistics ====" + wget --no-verbose https://download.geany.org/geany_git.tar.gz -O /tmp/geany_git.tar.gz + /venv/bin/python manage.py generate_i18n_statistics + rm -f /tmp/geany_git.tar.gz +fi + +# sync page contents from GIT back to the database +echo "==== Sync page contents ====" +/venv/bin/python /app/manage.py sync_pages + +# start the server +/venv/bin/python /app/manage.py runserver 0.0.0.0:8000
Modified: docker/irc_userlist 5 lines changed, 5 insertions(+), 0 deletions(-) =================================================================== @@ -0,0 +1,5 @@ +dummy-user-1 +dummy-user-2 +dummy-user-3 +dummy-user-4 +dummy-user-5
Modified: docker/local_settings.docker.py 124 lines changed, 124 insertions(+), 0 deletions(-) =================================================================== @@ -0,0 +1,124 @@ +# coding.: utf-8 + +# This file is exec'd from settings.py, so it has access to and can modify all +# the variables in settings.py. + +# If this file is changed in development, the development server will have to +# be manually restarted because changes will not be noticed immediately. + + +DEBUG = True + + +INTERNAL_IPS = ("127.0.0.1",) +ALLOWED_HOSTS = ('127.0.0.1', 'localhost') + +STATIC_ROOT = '/app/static' +MEDIA_ROOT = '/app/docker/data' + + +DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.sqlite3', + 'NAME': os.path.join(MEDIA_ROOT, 'geany_dev.db'), + }, + 'nightlybuilds': { + 'ENGINE': 'django.db.backends.sqlite3', + 'NAME': os.path.join(MEDIA_ROOT, 'geany_dev_nightlybuilds.db') + } +} + +SECRET_KEY = "f94b410d-0401-4f25-a69e-d5cbbf7717f3-1af070a9-ebef-4efe-a779-a9706bbdfd4a" +NEVERCACHE_KEY = "c8eb09b7-bd7e-4375-8293-6ffe53bf92e1-884d109e-6da4-4b99-bc56-075d8a498d4c" + +CSRF_COOKIE_HTTPONLY = True +CSRF_COOKIE_SECURE = False +SSL_FORCE_URL_PREFIXES = () +SESSION_COOKIE_SECURE = False +USE_X_FORWARDED_HOST = False + +CACHES = { + 'default': { + 'BACKEND': 'django.core.cache.backends.locmem.LocMemCache', + } +} + +FORCE_SCRIPT_NAME = '' +SERVER_EMAIL = 'root@localhost' +DEFAULT_FROM_EMAIL = 'root@localhost' + +NIGHTLYBUILDS_BASE_DIR = os.path.join(MEDIA_ROOT, 'nightly_mirror') +IRC_USER_LIST_FILE = '/data/irc_userlist' + +STATIC_DOCS_GEANY_SOURCE_TARBALL = '/tmp/geany_git.tar.gz' +STATIC_DOCS_GEANY_DESTINATION_DIR = os.path.join(MEDIA_ROOT, 'i18n') +STATIC_DOCS_GEANY_DESTINATION_URL = os.path.join(MEDIA_URL, 'i18n') +STATIC_DOCS_GEANY_I18N_STATISTICS_FILENAME = 'i18n_statistics.json' + +LOGGING = { + 'version': 1, + 'disable_existing_loggers': True, + 'formatters': { + 'verbose': { + 'format': '%(asctime)s %(name)s %(process)d %(threadName)s %(levelname)s %(message)s' + }, + }, + 'filters': { + 'require_debug_false': { + '()': 'django.utils.log.RequireDebugFalse' + }, + }, + 'handlers': { + 'console':{ + 'level':'DEBUG', + 'class':'logging.StreamHandler', + 'formatter': 'verbose' + }, + }, + 'loggers': { + '': { + 'handlers':['console'], + 'level':'DEBUG', + 'propagate': False, + }, + 'root': { + 'handlers':['console'], + 'level':'DEBUG', + 'propagate': False, + }, + 'py.warnings': { + 'handlers':['console'], + 'propagate': True, + 'level':'DEBUG', + }, + 'django': { + 'handlers':['console'], + 'propagate': True, + 'level':'DEBUG', + }, + 'django.db': { + 'handlers':['console'], + 'level': 'INFO', + 'propagate': True, + }, + 'django.template': { + 'handlers':['console'], + 'level': 'INFO', + 'propagate': True, + }, + 'django.utils.autoreload': { + 'handlers': [], + 'level': 'INFO', + 'propagate': True, + }, + 'MARKDOWN': { + 'level': 'INFO', + 'propagate': True, + }, + 'PIL': { + 'handlers': [], + 'level': 'INFO', + 'propagate': True, + }, + } +}
Modified: geany/settings.py 4 lines changed, 3 insertions(+), 1 deletions(-) =================================================================== @@ -594,7 +594,9 @@
# Instead of doing "from .local_settings import *", we use exec so that # local_settings has full access to everything defined in this module. -filename = os.path.join(PROJECT_APP_PATH, 'local_settings.py') # pylint: disable=invalid-name +# pylint: disable=invalid-name +local_settings_file_name = os.environ.get('LOCAL_SETTINGS_PY', 'local_settings.py') +filename = os.path.join(PROJECT_APP_PATH, local_settings_file_name) # pylint: disable=invalid-name if os.path.exists(filename): import sys from importlib.util import module_from_spec, spec_from_file_location
Modified: tox.ini 8 lines changed, 4 insertions(+), 4 deletions(-) =================================================================== @@ -32,7 +32,7 @@ commands = {envbindir}/pylint --rcfile=tox.ini {[tox]geany_modules}
[flake8] -exclude = build,.git,docs,migrations,local_settings.py +exclude = build,.git,docs,migrations,local_settings.py,local_settings.docker.py ignore = E127,E128, max-line-length = 100
@@ -46,12 +46,12 @@ sections = FUTURE,STDLIB,THIRDPARTY,FIRSTPARTY,LOCALFOLDER lines_after_imports = 2 from_first = true include_trailing_comma = true -skip = local_settings.py +skip = local_settings.py,local_settings.docker.py
-# the following secions are for pylint +# the following sections are for pylint [MASTER] ignore=.git -ignore-patterns=local_settings.py +ignore-patterns=local_settings.py,local_settings.docker.py persistent=no load-plugins=pylint_django
-------------- This E-Mail was brought to you by github_commit_mail.py (Source: https://github.com/geany/infrastructure).