diff --git a/CHANGELOG.md b/CHANGELOG.md index 42fcd66..d808ca2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,100 @@ -## v2.3.0 (Februrary 2017) +## v3.0.0 (June 2017) "The Unification" + +### Major Changes + +#### New Command for Theme Support in Build Container + +Added build container to run gulp in the primary theme when PLS is used. + +Usage: `docker-compose -f build.yml run --rm theme`. By default this runs gulp +without parameters, the equivalent of `npm run start`. + +View PL at http://theme..vm/pattern-lab/public + +Contribution by **Tanner** & **CJ** + +#### Merged generator-p2-env into generator-p2 + +New combined project contains the "master generator" as well as the environment +and infrastructure focused pieces of the generator stack. + +#### Vary Environment Setup by Production Hosting Destination + +Add hosting platform selection architecture with initial options: + * Default (Outrigger/Dev Cloud) + * Acquia Cloud + +#### Drop docker-compose.devcloud.yml + +While build.devcloud.yml is still with us, docker-compose.yml now has everything +needed for our "operational" services. On hosted environments you run as: + +```bash +# Start up the operational services for the QA environment of +DOCKER_ENV=qa docker-compose -p _qa up -d +# Run the Build CLI for the QA environment of +DOCKER_ENV=qa docker-compose -f build.devcloud.yml -p _qa run --rm cli +``` + +#### Node Version Change + +Switched to Node v6 by default instead of Node v4. + +### Environment Changes + +#### Things to Actively Use or Account For + +* Fixed errors where Drupal Console, Drush, and Grunt run without a further flag + or sub-command would error trying to process "bin/bash" as an argument. + (Solution by **CJ**) +* Added `composer` as a top-level service in the build container configuration. + Use with `docker-compose -f build.yml run --rm composer ` +* Switched Devtools mentions to Outrigger, including changing environment + variables such as $DEVTOOLS_SSH_KEY to $OUTRIGGER_SSH_KEY +* Changed directory structure of volume mounts to always use `/data///` + instead of `/data/_`. We hope this streamlines cleanup work. +* Add outrigger.project label to all containers, allowing commands such as: + `docker ps --filter "label=outrigger.project=" -a` to find all containers + and `docker ps --filter "label=outrigger.project=" -a -q | xargs docker stop` + to halt all of them. This is more comprehensive than a single `docker-compose ps` call. + +#### Things to Passively Benefit From + +* Minimize build.devcloud.yml and docker-compose.devcloud.yml by making the baseline + configuration a little more dynamic and using docker-compose "extends" as much + as possible. +* Added permissions fix to `src/sites/default` directory to keep it write-able + after a site install. This should prevent branch-switching permissions issues. + (Remember: Fix permissions problems by running /var/www/bin/fix-perms.sh in container.) +* If running grunt inside docker, `grunt git-setup` will configure your git + hooks to run via the docker build container. +* Switched to use the Outrigger Docker images. (This is a minimal change.) +* Added a persistent cache for Behat/Gherkin to the build container. +* Added clarifying terminal output about the un-usefulness of `--force` in + the build process. +* Centrally suppress grunt-drupal-tasks desktop notifications for build container. +* Fixed log shortcut in package.json to use docker-compose logs. + (A future release will remove this, as we do not encourage use of npm on local machine.) +* Fixed echo statement in start.sh executing and failing a stray grunt command. +* Added missing newlines from the end of several echo statements in start.sh + (Contribution by *Dave Murray*) +* Default the domain prompt to the previously confirmed projectName, rather than the + base of the current working directory (CWD). +* Remove vestigial bits of Phase2 from the environment prompts. + +### Usage Features + +* Run the generator via new Docker integration! This fixes the generator + upgrading/dependency chaos that many developers run into. + +### Technical Underbelly + +* Merged generator-p2-env into generator-p2. +* Upgraded to Yeoman 1.x and fixed related technical debt. +* Added Longjohn for vastly easier asynchronous and templating troubleshooting. +* Further race condition squashing. + +## v2.3.0 (February 2017) Various bugfixes and miscellaneous environment enhancements. @@ -6,6 +102,50 @@ Various bugfixes and miscellaneous environment enhancements. * Bumped version of [generator-gadget from v1.0.0-rc1 to v1.0.0](https://github.com/phase2/generator-gadget/releases/tag/v1.0.0) * Fixed redundant prompts not being streamlined. +### generator-p2-env changes + +This release was originally meant to be a bugfix release, but many great things +happened that postponed release. + +#### BASH History + +Persist BASH history in the build container across runs for a given project and +environment. + +This is in conjunction with a change to the build container, make sure to run +`docker pull phase2/devtools-build:php` first. + +#### Jenkins Overhaul + +Split Jenkins into its own sub-generator which can be separately used as +`yo p2-env:jenkins`. Further Jenkins-related changes: + +* Update Jenkins image to pull `latest` instead of the version released + alongside support for Docker 1.9. Recommend you pin your Jenkins image! +* Streamlined Jenkins jobs to use docker-compose environment variables instead + of the more verbose command-line flags. (COMPOSE_PROJECT_NAME and COMPOSE_FILE + instead of using the -f and -p flags all the time.) +* Do not re-run the seed-users script on update deploys. +* Added `drush cron` to the tasks that are run after site install or database + update on Jenkins deploy jobs. + +#### Other Changes + +* Explicitly declare that projects use Node 4 in build container. + This facilitates backwards compatibility if the build container switches to + Node 6 by default. +* Explicitly declare that Xdebug is disabled in build container to clarify + that it can be enabled. +* Shift docker-compose.yml and build.yml to use DOCKER_ENV environment variable + instead of hard-coding a local environment. This is a stepping stone to: + * Easily operate multiple local environment for the same project. + * Potential elimination of the separate build.devcloud.yml and + docker-compose.devcloud.yml files. +* Fixed networking with docker-compose schema v2 +* Fixed more edge cases with permissions on Drupal files directory. +* Fixed `bin/start.sh` would use empty DOCKER_ENV instead of defaulting to + `local` if you explicitly set your $DOCKER_ENV to an empty string. + ## v2.2.0 (January 2017) This release stabilizes some emergent bugs around Docker usage and composer builds with Drupal 8. @@ -13,13 +153,189 @@ This release stabilizes some emergent bugs around Docker usage and composer buil * Bumped version of [generator-p2-env to v1.2.0](https://bitbucket.org/phase2tech/generator-p2-env/src/master/CHANGELOG.md) * Bumped version of [generator-gadget to v1.0.0-rc1](https://github.com/phase2/generator-gadget/releases/tag/v1.0.0-rc1) +### generator-p2-env changes + +* Add build container support for Drupal Console (must include as a project + composer dependency.) Use with `docker-compose -f build.yml run --rm drupal`. +* Now using docker-compose manifest schema v2.1 +* Switched to use new DNSdock label configuration instead of environment + variable configuration in docker-compose manifests. +* Fixed `latest` not found for build container. (Drupal 7 now standardized on + PHP 5.6 and Drupal 8 on PHP 7.0) +* Streamlined `bin/start.sh` script + * Added `--rm` to more docker-compose commands in bin/start.sh to avoid + dangling containers. + * Using more of docker-compose tool's environment variables to minimize the + number of option flags needed. (e.g., COMPOSE_FILE, COMPOSE_PROJECT_NAME) + * Fixed site URI output at the end of execution. + ## v2.1.0 (July 2016) * Bumped version of [generator-p2-env to v1.1.0](https://bitbucket.org/phase2tech/generator-p2-env/src/master/CHANGELOG.md) * Pinned version of [generator-gadget to v0.5.0](https://github.com/phase2/generator-gadget/releases/tag/v0.5.0) * Pinned version of [generator-pattern-lab-starter to v2.0.0](https://github.com/phase2/generator-pattern-lab-starter) +### generator-p2-env changes (v1.1.0) + +* Added Docker container start and stop jobs for all environments. +* Added rebuild script to run `bin/start.sh with --update` for greater clarity + on already setup development environments. +* Added more Drupal 8 cache tables to default Drush excludes. + +### generator-p2-env changes (v1.0.0) + +#### Major Features + +##### Doctor.sh in the House + +There is now `bin/doctor.sh` to check on project setup. If your project is not +working, please run it to confirm easily checked problems are not the issue. +Parts of this might later move into Grunt Drupal Tasks. + +##### Operational vs. Base Build Container + +All build containers previously extended from `base`, a parent container that +sets up common environment variables, volume mounts, and other behaviors. +A new `operational` service definition is now the direct parent of the +functional containers. + +This allows projects to define new build services that extend from `operational` + when they need the use of backing services like the database, or extend from +`base` when they do not. This allows pure filesystem or build operations to be +run without the additional overhead. + +#### Compatibility Breaks + +* `base` build container no longer has dependency on backing services like + database or memcache. This means it also cannot run commands accessing those + services. +* New Drupal 7 projects or updates via the `--replay` will now use PHP 5.6. + This can be manually changed. + +#### Release Notes + +##### Features + +* Easily create new build services that do or do not use the Drupal backing services. +* Added `npm run d` as a convenient shorthand to run `docker-compose -f build.yml run` + in your local environment (not inside container). Use commands like `npm run d cli` + to start a "bash" session in the build container. +* Added `npm run logs` to watch a live stream of PHP logs, containing Drupal syslog + if that module is active. This log is not included in the `docker logs` output. +* Boosted default PHP max_execution_time to 60 seconds. +* Added output of Drupal site URL to `bin/start.sh` output. +* Added `bin/doctor.sh` to automate checks on project setup. +* Switched to new [Docker image naming for Apache/PHP and Varnish containers](https://hub.docker.com/r/phase2/apache-php/). +* Updated Drupal 7 projects to use PHP 5.6. + ## v2.0.0 (April 2016) * Bumped version of [generator-p2-env to v0.3.0](https://bitbucket.org/phase2tech/generator-p2-env/src/master/CHANGELOG.md) * Added test coverage + +### generator-p2-env changes + +This release represents a significant amount of R&D pursued as part of field-use +of the Yo P2 generator suite. After some shakedown and some light structuring +changes, this is likely close to a 1.0 release. + +#### Major Features + +##### Exception Handling with Bash Traps + +Jenkins jobs and the `bin/start.sh` script now use signal traps to catch errors or +interruptions of any kind. When that happens the script stops execution and the +Docker containers are taken down. You can suppress errors on a line-by-line basis +in many grunt commands with the `--force` flag, or in Bash by simply appending +`|| true` to the end of the line. This fixes the problem of Jenkins reporting +false success. + +##### Simplification of docker-compose Configuration Files + +Docker Compose is a utility that bridges complex configuration manifests to the +Docker client for management of Docker containers. Previously, we needed two +files per environment: one for operating services, the other for build targets. + +Now there are two sets of these files (leaving aside the one for Jenkins): + +* `docker-compose.yml` and `build.yml` for local development +* `docker-compose.devcloud.yml` and `build.devcloud.yml` for environments hosted in Dev Cloud. + +(The dependency on Dev Cloud is one of filesystem conventions, it is not tied +to Phase2 infrastructure specifically.) + +Running docker-compose commands locally is unchanged. However, on ci.p2devcloud.com, +it has a significant change. + +> ###### No Longer +> +> ``` +> docker-compose -f build.qa.yml -p project_qa run drush cache-clear all +> ``` + +> ###### Instead +> +> ``` +> DOCKER_ENV=qa docker-compose -f build.devcloud.yml -p project_qa run drush cache-clear all +> ``` + +**This allows us to spin up new environments on Dev Cloud as easily as creating a new Jenkins job.** + +In the course of this change, the docker-compose files were also adjusted to have more inline +documentation and intra-file inheritance. This allows much easier management of consistent volume +mounts, environment variables, and other pieces of configuration used in common. + +##### `bin/start.sh` a More Robust Tool + +This script is used in all Jenkins deployment and CI jobs to manage container +setup, build process, and site installation in a consistent manner. It is also +the recommended approach to start up a project locally for the first time, or +when a clean reset is needed. + +The `bin/start.sh` script has got command-line options now, including: + +* `--update` to skip installation +* `--no-validate` to skip static analysis in build +* `--noop` to simulate script execution +* `--help` learn what else it can do! + +#### Compatibility Breaks + +* docker-compose commands on Dev Cloud are changed, see long description above. +* Rename `ms` environment to `review` for clarity. +* Convert URLs to the form qa-project.ci.p2devcloud.com for easier DNS & SSL management +* Default new projects to use the ci2.p2devcloud.com host. Use `--ciHost=ci.p2devcloud.com` + to keep using the original. +* Project-specific Drush volume mount is now placed inside the container at `/root/.drush` +* Drop support for environment variable `DEVTOOLS_PRIVATE_KEY`. SSH keys are now + only injected into the Jenkins container with volume mounts. If your local + `id_rsa` key does not private the necessary access for local Jenkins use, temporarily + edit the `jenkins.yml` file. In the future this might be parameterized + (see [Docker Compose #2441](https://github.com/docker/compose/issues/2441)) + +#### Release Notes + +##### Features + +* Jenkins deploy jobs now have an option to run update instead of reinstall. +* start.sh now takes real options and has usage output. +* Make better use of error handling to halt and clean up after errors in jobs. +* Add volume mounts to persist tool caches across build runs. +* Added optional MailHog service to capture Drupal email and facilitate testing. + **See generated `TODOS.md`, this feature requires manual action per project.** +* Streamline docker-compose manifests, `*.devcloud.yml` instead of one per environment. +* Break Jenkins home into views by environment with primary for key shortcuts. +* Add jenkins-test-fail job to check on Jenkins error handling. +* Added deploy-local job for local testing of Jenkins +* Added the colorization library from _devtools_vm past, began light use. +* Automated test coverage, 45 assertions and counting. +* Generated documentation improvements + +##### Bugs + +* Stop clobbering registry-rebuild command which is provided by the devtools-build container +* Stop risky use of nginx-proxy reserved `VIRTUAL_HOST` environment variable in build containers. +* Fixed generator `--replay` run if `flowdockApiKey` and `gitRepoUrl` are not set. +* Replace each instance of hyphen with underscore to generate machine name. +* Make shell scripts executable by default and remove explicit bash shelling to execute. +* Use underscores in lieu of hyphens in all docker-centric identifiers. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6378672..f272b8c 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,31 +1,20 @@ # CONTRIBUTING +## New Feature Ideas + +Reach out by creating a new issue to discuss features if your idea might best be +solved in one of the other generator projects, or as part of a different project +in the Outrigger suite. + ## Run the Tests Before submitting a PR, please run the tests in this repository to confirm nothing has broken. -```bash -npm test -``` - -or - ```bash docker-compose run --rm cli npm test ``` -## Docker-based Development - -For development purposes, or sometimes ease of version management, this repo -provides Docker integration. Clone git to the version you want and run from -inside the repo: - -```bash -docker-compose run --rm cli npm install -YO_PROJECT_DIRECTORY=path/to/empty/directory docker-compose run --rm p2 -``` - ## Testing Branches without Docker If you would like to test out changes to generator-p2 or one of the above diff --git a/LICENSE.txt b/LICENSE.txt new file mode 100644 index 0000000..f1b9ff6 --- /dev/null +++ b/LICENSE.txt @@ -0,0 +1,21 @@ +The MIT License (MIT) + +Copyright (c) 2017 Phase2 + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/README.md b/README.md index b2ec3cb..17d90d3 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Yo P2! – the Phase2 Site Generator +# Yo P2! � the Phase2 Site Generator > Yeoman generator that weaves together multiple generators, tools, and Phase2 best practices to kick off Drupal projects in style. @@ -9,40 +9,31 @@ aspect of the scaffolding. As the name implies, this generator focuses on Phase2 practices, where some of the other generators in use are focused on broader industry best practices. -## Installation +## Installation & Usage -1. Identify the latest release by checking the latest tag in this repo. -2. Run the following command, replacing {version} with the tag name you wish to install. +### Using Docker -```bash -npm install --global git+ssh://bitbucket.org/phase2tech/generator-p2.git#{version} -``` - -## Run the Generator - -> If you have trouble getting the generator to run, or updating the generator, -check out the [CONTRIBUTING.md](./CONTRIBUTING.md) guide for the instructions on -how to use Docker to run the generator. - -Make an empty folder, and initiate the generator: +Docker-based usage is available as of v3.0.0. It is the recommended approach as +updates tend to be less prone to complication. ```bash -yo p2 +git clone git@bitbucket.org:phase2tech/generator-p2.git +cd generator-p2 +git checkout v3.0.0 +docker-compose run --rm cli npm install +mkdir ~/path/to/empty/directory +YO_PROJECT_DIRECTORY=~/path/to/empty/directory docker-compose run --rm yo p2 ``` -Once all questions are answered a complete code scaffolding will be created. -Generate a working codebase with another one-liner: +### Using npm ```bash -grunt +npm install --global git+ssh://bitbucket.org/phase2tech/generator-p2.git#v3.0.0 +mkdir ~/path/to/empty/directory +cd ~/path/to/empty/directory +yo p2 ``` -> **Note that if you opted-in to create a Docker-based environment you will have -alternate instructions, you will not be running grunt.** - -> ***Please read the output at the end of the generator run, it has instructions -on next steps.*** - ### Command-line Options * **`--replay`**: Re-run the generator against an existing project, using previously @@ -53,7 +44,59 @@ grunt-drupal-tasks. generation process. (Applied by default when you opt-in for the Docker-based Phase2 Environment.) -## Features +## Answering Yeoman Prompts + +### First-time Usage + +Once Yeoman has been started, you will be prompted to answer six to a dozen +questions about your project. These questions are used by the generator to +select templates, create variables for use in the templates, and further execute +any code or customizations on top of the templated scaffolding. + +> ***Please read the output at the end of the generator run, it has instructions +on next steps.*** + +### Running on a Generated Project + +When you use this to generate a project, all your answers are recorded by Yeoman +in a .yo-rc.json file at the root of your repository. As long as you commit this +to the codebase, from then if you run the generator targeting that directory it +will use the previous answers as the defaults to every prompt. For even faster +usage, you can run `yo p2 --replay` and it will automatically run the generator +without stopping to ask old questions. + +If you run the generator, it will overwrite files. That's the idea in this case, +as your goal is to cherrypick good new stuff the generator provides into your +project. Run `git add -p .` to view each change one chunk at a time, and decide +if it makes sense to discard or keep. Feel free to ask for help. + +The extensive changelog notes are intended to help guide this process. + +### Retrofitting an Existing Project + +If you have an existing project and want to retrofit it to use the scaffolding +and tools this generator provides, there is no magical solution. + +1. Create a new empty project, answer the prompts to align as closely as +possible with your current projects practices and the goals you have in using +the generator. +2. Copy all the pieces of your codebase into the new structure, including your +.git directory. +3. Adjust your scripts to the new structure, and adjust the generated +configuration and scripts to the parts of your codebase that are not changing. + +## Next Steps + +Read the generated TODOS.md file. Remove things as you work through mastering +and running your vanilla Drupal system for the first time. + +If you are not using the Docker-based development environment, you can get your +codebase built by running `grunt`. From there you are on your own. + +If you are using the Docker-based development environment, run `./bin/start.sh` +and your site will be built and a new site installed. + +## Child Generators ### Backend Build Process, Testing, & Task Runner @@ -65,7 +108,7 @@ A best practice setup of GDT with a few extra goodies are provided by Yo P2's use of [Generator Gadget](https://github.com/phase2/generator-gadget) to produce the top-level Drupal application scaffolding. -### Best Practices Frontend +### Component-driven Frontend [Pattern Lab Starter]([Pattern Lab Starter](http://git.io/p2pls)) is used to provide an optional Drupal theme with Pattern Lab for collaborative, @@ -78,13 +121,36 @@ so a full site compile can happen by running the project's default build task: ` ### Docker-based Development Environment -We integrate with Phase2 DevTools (now http://outrigger.sh) to provide a quick, -consistent, container-based local development environment that can also be used -on Phase2's centralized docker hosts to provide testing and review environments. +We integrate with Outrigger (now http://outrigger.sh) to provide a quick, +consistent, container-based, local development environment that can also be used +on Phase2's centralized Outrigger hosting to provide integration, testing, and +other environments. This uses a sub-generator of generator-p2 which can be separately executed as `yo p2:environment`. +#### Feature Breakdown + +* `docker-compose.yml` which provides Docker configuration for the "operational" +services needed to run your Drupal site locally. + * Default support for Apache and MariaDB containers, with optional support + for Redis or Memcache, Varnish, and Mailhog + * Supports Outrigger's DNSDock-based DNS service for nice URLs. + * Supports nginx-proxy for Outrigger Hosting nice URLs. +* `build.yml` which provides tailored use of the Outrigger Build container. + * Persists command-line history on a per project/per environment basis. + * Persists build tool cache for Drush, Behat/Gherkin, Drupal Console, npm, + composer, yarn, bower, fontconfig, and any other tool which uses ~/.cache. + * Defines easy-access use of grunt, drush, drupal console, and composer. + * Defines a "CLI" service to open a standing BASH session with the build + container. +* Default settings.php configuration that sets up Drupal environment +configuration the "Docker way" to work out-of-the-box. +* Default Drush configuration for slim database exports and a nice project alias. +* Use Grunt Drupal Tasks via Docker, using minor configuration adjustments. +* Provides a range of shell scripts to run builds, verify aspects of the system, +create default users, and more. + ### Project-specific CI & Environment Management If you liked the Docker-based environments, you can also use our Docker-based @@ -92,4 +158,14 @@ Jenkins instance to manage your central test environments, complete with default jobs so you can start continuous integration with zero further configuration. This uses a sub-generator of generator-p2 which can be separately executed as -`yo p2:jenkins`. \ No newline at end of file +`yo p2:jenkins`. + +### Feature Breakdown + +* Jenkins 1.x CI server pre-configured with various jobs: + * CI job polling your git repo to run tests. + * "dev-support" job to produce nightly database dumps of a fresh site install. + * Optional Environment lifecycle management jobs for Development, QA, and Review + environments, including: start/stop containers, deploy code, reset admin user password + * Test jobs to confirm the system is working. +* Workspace details persisted in volume mounts inside repo:jenkins/env/workspace. diff --git a/package.json b/package.json index 18a7c80..c823327 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "generator-p2", - "version": "3.0.0-pre", + "version": "3.0.0", "description": "Phase2 Yeoman generator", "license": "MIT", "main": "app/index.js", @@ -27,8 +27,8 @@ ], "dependencies": { "chalk": "^1.0.0", - "generator-gadget": "git+https://git@github.com/phase2/generator-gadget.git#067efedb9468b8d7b6cc48ceeabc5e18c1842bcf", - "generator-pattern-lab-starter": "^3.0.2", + "generator-gadget": "^1.1", + "generator-pattern-lab-starter": "git+https://git@github.com/phase2/generator-pattern-lab-starter.git#a626d763d9619228de3ea56c1ee9b804ff7ef131", "lodash": "^4.17.4", "semver": "^5.3.0", "yeoman-generator": "^1.1.0",