Automating Drupal Migration with Shell Scripts

2023-09-01

The challenge I faced during one migration was that I had to run some standard migration scripts, then execute some manual Drush commands, and then continue with automated scripts. In addition, I wanted to exclude some migration steps as well. Doing all this manually is very laborious, especially if it has to be run more than once. So I came up with the scripts that generate scripts to help automate the migration a bit.

Requirements

All Drupal migrations contain two parts: configuration and content. We will define the automation for both parts. This tutorial assumes that the Drupal 9/10 site was set up with two databases, as required for all migrations. The Drupal 7 files were placed into a folder sites/default/files/d7-export and the Migrate Plus module was installed.

Generate Migration Configuration

First, we'll use the Migrate Plus module to generate a migration configuration, which will include a full set of migration routines.

drush migrate-upgrade --legacy-db-key=migrate --legacy-root=sites/default/files/d7-export --configure-only

Migrating Configuration

Then, we'll create a configuration migration script with the name migration-config-01.sh. Inside, we'll place the code that will generate the next shell script file for us. The code in the file we are creating will do the following:

Get a list of configuration migrations from Drush
Run through the list and exclude items from the EXCLUDE_STEPS list
It will generate a shell script with a name stored in NEXT_MIGRATION_SCRIPT that will contain a list of migrations that we want to run
The generated shell script will also contain a script that will export the database after the migrations are run, so we have a clear restore point to return to if the following steps fail
Then the current script will execute the newly generated file, migrating the configuration

#!/bin/sh
# Generate the config scripts to import structure
MIGRATION_STEPS=migration-config-steps.txt
CURRENT_MIGRATION_SCRIPT=migration-config-01.sh
NEXT_MIGRATION_SCRIPT=migration-config-02.sh
EXCLUDE_STEPS=(
  'upgrade_d6_book_settings'
)
# Log a message to the screen, so we can keep track where we are
echo "[sh] starting $CURRENT_MIGRATION_SCRIPT"
# Get a list of conifguration migrations to run from Drush
drush migrate-status --tag=Configuration --group="migrate_drupal_7" --fields="id" --format="string" > $MIGRATION_STEPS
# Reset the next script file, if it exist, or create a new one
echo "#!/bin/sh" > $NEXT_MIGRATION_SCRIPT
echo "echo \"[sh] starting $NEXT_MIGRATION_SCRIPT\"" >> $NEXT_MIGRATION_SCRIPT
# Go through the list of migrations and generate a script to run them all
# but exclude the ones that are on the exclude list
while IFS=$' \t\r\n' read -r line; do
  if echo "${EXCLUDE_STEPS[*]}" | grep -qw "$line"; then
    echo "[sh] excluded $line from migration"
    continue
  else
    # Write an echo message and an import script to a new file
    echo "echo \"[sh] starting $line\"" >> $NEXT_MIGRATION_SCRIPT
    echo "drush migrate:import --execute-dependencies $line" >> $NEXT_MIGRATION_SCRIPT
  fi
done < $MIGRATION_STEPS
echo "# Make a database backup to return to this point" >> $NEXT_MIGRATION_SCRIPT
echo "mysqldump -u YourUser -p YourDatabaseName > d9-migrated-config.sql" >> $NEXT_MIGRATION_SCRIPT
echo "echo \"[sh] completed $NEXT_MIGRATION_SCRIPT\"" >> $NEXT_MIGRATION_SCRIPT
echo "./migration-content-01.sh" >> $NEXT_MIGRATION_SCRIPT
echo "[sh] completed $CURRENT_MIGRATION_SCRIPT"
./$NEXT_MIGRATION_SCRIPT

Migrating Content

First we'll create a content migration script, with a name migration-content-01.sh. Inside we'll place the code that will generate the next shell script file for us. The code in the file we are creating will do the following:

Get a list of content migrations from Drush
Run through the list and exclude items from the EXCLUDE_STEPS list
It will generate a shell script with a name stored in NEXT_MIGRATION_SCRIPT that will contain a list of migrations that we want to run
The generated shell script will also contain a script that will export the database after the migrations are run, so we have a clear restore point to return to if the following steps fail
Then the current script will execute the newly generated file, migrating the configuration

#!/bin/sh
CURRENT_MIGRATION_SCRIPT=migration-content-01.sh
MIGRATION_STEPS=migration-content-steps.txt
NEXT_MIGRATION_SCRIPT=migration-content-02.sh
EXCLUDE_STEPS=(
  'upgrade_d7_menu_links_user'
  'upgrade_d7_menu_links'
  'upgrade_d7_menu_links_node_page'
  'upgrade_d7_menu_links_node_webform'
  'upgrade_d7_taxonomy_term_tags'
  'upgrade_d7_taxonomy_term_article_categories'
  'upgrade_d7_url_alias'
)
# Log a message to the screen, so we can keep track where we are
echo "[sh] starting $CURRENT_MIGRATION_SCRIPT"
# Get a list of migrations to run
drush migrate-status --tag=Content --group="migrate_drupal_7" --fields="id" --format="string" > $MIGRATION_STEPS
# Reset the next script file to 0 if it exists
echo "#!/bin/sh" > $NEXT_MIGRATION_SCRIPT
echo "echo \"[sh] starting $NEXT_MIGRATION_SCRIPT\"" >> $NEXT_MIGRATION_SCRIPT
# Go through the list of migrations and generate a script to run them all
while IFS=$' \t\r\n' read -r line; do
  if echo "${EXCLUDE_STEPS[*]}" | grep -qw "$line"; then
    echo "excluded $line from migration"
    continue
  else
    # Write an echo message and an import script to a new file
    echo "echo \"[sh] starting $line\"" >> $NEXT_MIGRATION_SCRIPT
    echo "lando drush migrate:import --execute-dependencies $line" >> $NEXT_MIGRATION_SCRIPT
  fi
done < $MIGRATION_STEPS
echo "# Make a backup to return to" >> $NEXT_MIGRATION_SCRIPT
echo "mysqldump -u YourUser -p YourDatabaseName >  d9-migrated-content-04.sql" >> $NEXT_MIGRATION_SCRIPT
echo "echo \"[sh] completed $NEXT_MIGRATION_SCRIPT=\"" >> $NEXT_MIGRATION_SCRIPT
echo "[sh] site migration is complete now, start content checks" >> $NEXT_MIGRATION_SCRIPT
echo "[sh] completed $CURRENT_MIGRATION_SCRIPT"
./$NEXT_MIGRATION_SCRIPT

This is just a basic framework. Hopefully this gives you a good start into automating your Drupal migration.