diff --git a/.gitignore b/.gitignore index 802aab9..77b24eb 100644 --- a/.gitignore +++ b/.gitignore @@ -1,78 +1,79 @@ # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *$py.class # C extensions *.so # Distribution / packaging .Python env/ build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ *.egg-info/ .installed.cfg *.egg # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec # Installer logs pip-log.txt pip-delete-this-directory.txt +sentry_install_log*.txt # Unit test / coverage reports htmlcov/ .tox/ .coverage .coverage.* .cache nosetests.xml coverage.xml *.cover .hypothesis/ # Translations *.mo *.pot # Django stuff: *.log local_settings.py # Sphinx documentation docs/_build/ # PyBuilder target/ # Ipython Notebook .ipynb_checkpoints # pyenv .python-version # https://docs.docker.com/compose/extends/ docker-compose.override.yml *.tar data/ .vscode/tags # custom Sentry config sentry/sentry.conf.py sentry/config.yml sentry/requirements.txt diff --git a/README.md b/README.md index 2a0267c..f74fbaa 100644 --- a/README.md +++ b/README.md @@ -1,55 +1,55 @@ # Sentry 10 On-Premise BETA [![Build Status][build-status-image]][build-status-url] Official bootstrap for running your own [Sentry](https://sentry.io/) with [Docker](https://www.docker.com/). **NOTE:** If you are not installing Sentry from scratch, our recommendation is to visit [On-Premise Stable for Sentry 9.1.2](https://github.com/getsentry/onpremise/tree/stable) as this version may not be fully backward compatible. If you still want to try it out make sure you are on 9.1.2 first, back up your old Docker volumes just in case, and remember that if you haven't set up Redis persistency yourself some of your data (like your stats) may be lost during the upgrade. ## Requirements * Docker 17.05.0+ * Compose 1.19.0+ ## Minimum Hardware Requirements: * You need at least 2400MB RAM ## Setup To get started with all the defaults, simply clone the repo and run `./install.sh` in your local check-out. There may need to be modifications to the included example config files (`sentry/config.example.yml` and `sentry/sentry.conf.example.py`) to accommodate your needs or your environment (such as adding GitHub credentials). If you want to perform these, do them before you run the install script and copy them without the `.example` extensions in the name (such as `sentry/sentry.conf.py`) before running the `install.sh` script. The recommended way to customize your configuration is using the files below, in that order: * `config.yml` * `sentry.conf.py` * `.env` w/ environment variables We currently support a very minimal set of environment variables to promote other means of configuration. -If you have any issues or questions, our [Community Forum](https://forum.sentry.io/c/on-premise) is at your service! +If you have any issues or questions, our [Community Forum](https://forum.sentry.io/c/on-premise) is at your service! Everytime you run the install script, it will generate a log file, `sentry_install_log-.txt` with the output. Sharing these logs would help people diagnose any issues you might be having. ## Event Retention Sentry comes with a cleanup cron job that prunes events older than `90 days` by default. If you want to change that, you can change the `SENTRY_EVENT_RETENTION_DAYS` environment variable in `.env` or simply override it in your environment. If you do not want the cleanup cron, you can remove the `sentry-cleanup` service from the `docker-compose.yml`file. ## Securing Sentry with SSL/TLS If you'd like to protect your Sentry install with SSL/TLS, there are fantastic SSL/TLS proxies like [HAProxy](http://www.haproxy.org/) and [Nginx](http://nginx.org/). You'll likely want to add this service to your `docker-compose.yml` file. ## Updating Sentry -The included `install.sh` script is meant to be idempotent and to bring you to the latest version. What this means is you can and should run `install.sh` to upgrade to the latest version available. +The included `install.sh` script is meant to be idempotent and to bring you to the latest version. What this means is you can and should run `install.sh` to upgrade to the latest version available. Remember that the output of the script will be stored in a log file, `sentry_install_log-.txt`, which you may share for diagnosis if anything goes wrong. ## Resources * [Documentation](https://docs.sentry.io/server/installation/docker/) * [Bug Tracker](https://github.com/getsentry/onpremise/issues) * [Forums](https://forum.sentry.io/c/on-premise) * [IRC](irc://chat.freenode.net/sentry) (chat.freenode.net, #sentry) [build-status-image]: https://api.travis-ci.com/getsentry/onpremise.svg?branch=master [build-status-url]: https://travis-ci.com/getsentry/onpremise diff --git a/install.sh b/install.sh index 742d131..9d6e32f 100755 --- a/install.sh +++ b/install.sh @@ -1,161 +1,165 @@ #!/usr/bin/env bash set -e +# Thanks to https://unix.stackexchange.com/a/145654/108960 +log_file="sentry_install_log-`date +'%Y-%m-%d_%H-%M-%S'`.txt" +exec &> >(tee -a "$log_file") + MIN_DOCKER_VERSION='17.05.0' MIN_COMPOSE_VERSION='1.19.0' MIN_RAM=2400 # MB SENTRY_CONFIG_PY='sentry/sentry.conf.py' SENTRY_CONFIG_YML='sentry/config.yml' SENTRY_EXTRA_REQUIREMENTS='sentry/requirements.txt' DID_CLEAN_UP=0 # the cleanup function will be the exit point cleanup () { if [ "$DID_CLEAN_UP" -eq 1 ]; then return 0; fi echo "Cleaning up..." docker-compose stop &> /dev/null DID_CLEAN_UP=1 } trap cleanup ERR INT TERM echo "Checking minimum requirements..." DOCKER_VERSION=$(docker version --format '{{.Server.Version}}') COMPOSE_VERSION=$(docker-compose --version | sed 's/docker-compose version \(.\{1,\}\),.*/\1/') RAM_AVAILABLE_IN_DOCKER=$(docker run --rm busybox free -m 2>/dev/null | awk '/Mem/ {print $2}'); # Compare dot-separated strings - function below is inspired by https://stackoverflow.com/a/37939589/808368 function ver () { echo "$@" | awk -F. '{ printf("%d%03d%03d", $1,$2,$3); }'; } # Thanks to https://stackoverflow.com/a/25123013/90297 for the quick `sed` pattern function ensure_file_from_example { if [ -f "$1" ]; then echo "$1 already exists, skipped creation." else echo "Creating $1..." cp -n $(echo "$1" | sed 's/\.[^.]*$/.example&/') "$1" fi } if [ $(ver $DOCKER_VERSION) -lt $(ver $MIN_DOCKER_VERSION) ]; then echo "FAIL: Expected minimum Docker version to be $MIN_DOCKER_VERSION but found $DOCKER_VERSION" exit -1 fi if [ $(ver $COMPOSE_VERSION) -lt $(ver $MIN_COMPOSE_VERSION) ]; then echo "FAIL: Expected minimum docker-compose version to be $MIN_COMPOSE_VERSION but found $COMPOSE_VERSION" exit -1 fi if [ "$RAM_AVAILABLE_IN_DOCKER" -lt "$MIN_RAM" ]; then echo "FAIL: Expected minimum RAM available to Docker to be $MIN_RAM MB but found $RAM_AVAILABLE_IN_DOCKER MB" exit -1 fi # Clean up old stuff and ensure nothing is working while we install/update docker-compose down --rmi local --remove-orphans echo "" echo "Creating volumes for persistent storage..." echo "Created $(docker volume create --name=sentry-data)." echo "Created $(docker volume create --name=sentry-postgres)." echo "Created $(docker volume create --name=sentry-redis)." echo "Created $(docker volume create --name=sentry-zookeeper)." echo "Created $(docker volume create --name=sentry-kafka)." echo "Created $(docker volume create --name=sentry-clickhouse)." echo "Created $(docker volume create --name=sentry-symbolicator)." echo "" ensure_file_from_example $SENTRY_CONFIG_PY ensure_file_from_example $SENTRY_CONFIG_YML ensure_file_from_example $SENTRY_EXTRA_REQUIREMENTS echo "" echo "Generating secret key..." # This is to escape the secret key to be used in sed below SECRET_KEY=$(head /dev/urandom | tr -dc "a-z0-9@#%^&*(-_=+)" | head -c 50 | sed -e 's/[\/&]/\\&/g') sed -i -e 's/^system.secret-key:.*$/system.secret-key: '"'$SECRET_KEY'"'/' $SENTRY_CONFIG_YML echo "Secret key written to $SENTRY_CONFIG_YML" echo "" echo "Building and tagging Docker images..." echo "" # Build the sentry onpremise image first as it is needed for the cron image docker-compose pull --ignore-pull-failures docker pull ${SENTRY_IMAGE:-getsentry/sentry:latest} docker-compose build --force-rm web docker-compose build --force-rm echo "" echo "Docker images built." # Very naively check whether there's an existing sentry-postgres volume and the PG version in it if [[ $(docker volume ls -q --filter name=sentry-postgres) && $(docker run --rm -v sentry-postgres:/db busybox cat /db/PG_VERSION 2>/dev/null) == "9.5" ]]; then docker volume rm sentry-postgres-new || true # If this is Postgres 9.5 data, start upgrading it to 9.6 in a new volume docker run --rm \ -v sentry-postgres:/var/lib/postgresql/9.5/data \ -v sentry-postgres-new:/var/lib/postgresql/9.6/data \ tianon/postgres-upgrade:9.5-to-9.6 # Get rid of the old volume as we'll rename the new one to that docker volume rm sentry-postgres docker volume create --name sentry-postgres # There's no rename volume in Docker so copy the contents from old to new name # Also append the `host all all all trust` line as `tianon/postgres-upgrade:9.5-to-9.6` # doesn't do that automatically. docker run --rm -v sentry-postgres-new:/from -v sentry-postgres:/to alpine ash -c \ "cd /from ; cp -av . /to ; echo 'host all all all trust' >> /to/pg_hba.conf" # Finally, remove the new old volume as we are all in sentry-postgres now docker volume rm sentry-postgres-new fi echo "" echo "Setting up database..." if [ $CI ]; then docker-compose run --rm web upgrade --noinput echo "" echo "Did not prompt for user creation due to non-interactive shell." echo "Run the following command to create one yourself (recommended):" echo "" echo " docker-compose run --rm web createuser" echo "" else docker-compose run --rm web upgrade fi SENTRY_DATA_NEEDS_MIGRATION=$(docker run --rm -v sentry-data:/data alpine ash -c "[ ! -d '/data/files' ] && ls -A1x /data | wc -l || true") if [ "$SENTRY_DATA_NEEDS_MIGRATION" ]; then echo "Migrating file storage..." # Use the web (Sentry) image so the file owners are kept as sentry:sentry docker-compose run --rm --entrypoint /bin/bash web -c \ "mkdir -p /tmp/files; mv /data/* /tmp/files/; mv /tmp/files /data/files" fi echo "Boostrapping Snuba..." docker-compose up -d kafka redis clickhouse until $(docker-compose run --rm clickhouse clickhouse-client -h clickhouse --query="SHOW TABLES;" | grep -q sentry_local); do # `bootstrap` is for fresh installs, and `migrate` is for existing installs # Running them both for both cases is harmless so we blindly run them docker-compose run --rm snuba-api bootstrap --force || true; docker-compose run --rm snuba-api migrate || true; done; echo "" set -o allexport source .env set +o allexport echo "Migrating old events for the last $SENTRY_EVENT_RETENTION_DAYS days..." docker-compose run --rm web django backfill_eventstream --no-input --last-days $SENTRY_EVENT_RETENTION_DAYS echo "" cleanup echo "" echo "----------------" echo "You're all done! Run the following command to get Sentry running:" echo "" echo " docker-compose up -d" echo ""