In previous posts, we have provided tips and tricks for a successful Drupal migration from earlier versions of the content management system to the latest Version 8. In this installment of our series, we will talk about the Drupal 8 Migrate Plugin System.
Series Mapping:
Part 1: Migration Setup & Mappings
Part 2: Working with Processors
Part 3: Coming soon!
About Drupal 8 Migrate
The migrate module has been moved into the core in Drupal 8, showing the community dedicated to making the process of upgrading between versions or migrating into Drupal an easier path to a successful move. The migrate module takes advantage of the Drupal 8 Plugin system, offering developers several Plugin types that they can implement: MigrateProcessPlugin, MigrateSourcePlugin, MigrateDestinationPlugin.
In an earlier blog post in this series, we dove into an introduction of the Migrate module in Drupal 8 and reviewed a basic setup of the migration mappings needed to get started. In this post, we will dive into MigrateProcessPlugins: what they are, how they work and examples on how to create your own.
What is a Drupal 8 Migrate Processor?
A migration processor is a plugin that is used to manipulate data that is being mapped from a source to a destination. These plugins are typically small, but very powerful. Processors are used in the “process” portion of your migration configuration file.
Drupal 8 comes with several migration processors out-of-the-box. Some of the more notable ones that you will likely use on a regular basis are:
- get – Default, 1:1 data migration plugin
- default_value – Allows you to define a default value for a field
- explode – Converts a string into an array of strings based on a delimiter
- iterator – Iterates over an array of values to perform a process on
- migration_lookup – Looks up an entity based on an ID from a source to a destination
- skip_on_empty – Skips the current field or row if the value is empty in the migration
What Are the Benefits of Using a Drupal 8 Migrate Processor?
Migration process plugins provide developers with more flexibility and reusability when working with migrations in Drupal 8. By utilizing the Drupal 8 Plugin system, the Migrate module allows users to create new processors that implement a generic interface and can essentially be plug and play. In addition, the OO design of the plugin system allows developers to utilize inheritance of abstract classes and enforcement of methods if they have several variations of a plugin, without reinventing the wheel or duplicating code every time.
In English, this means that I can develop my own processors that manipulate data from my data source in any way that I want, and then reuse it across any migration that needs to use it.
How Does a MigrateProcessPlugin Work?
If you are finding that the out-of-the-box MigrateProcessPlugin’s are not enough for your use case, you may want to consider creating a new custom MigrateProcessPlugin. Creating a new process plugin in Drupal 8 is a fairly simple task.
Let’s take a look at a simple MigrateProcessPlugin as an example to get us started.
In the DefaultValue MigrateProcessPlugin, we only have one method that is implemented, called “transform”. Transform takes in a value that is passed in by the user, checks to see if it is set, and sets that value for the field in the destination mapping.
How Do I Use a Process Plugin in My Migration?
Using a process plugin in your migration is relatively simple. If you have written a migration before, you have used these without even knowing it. Let’s take a look at the example below:
In this example, you will notice several plugins are utilized:
- get: The “get” plugin is the default plugin. Any mapping that does not have a child element defining a plugin will utilize the “get” plugin
- default_value: In this example, we are setting the user ID of a blog_post to user 1
When am I Going to Need a Custom Processor Plugin?
Before you venture down the path of creating custom processors, ask yourself the following questions:
- Can any of the existing process plugins, or a combination of any of the existing process plugins, manipulate the data to the format I need?
- Is the data transformation something that follows a pattern?
- Is this processing/data manipulation something that will need to be used for more than one migration?
- Is this a transformation that other users/migrations may benefit from?
If the answer to any of the questions above is “YES”, it is worth your time to create a custom processor plugin.
Stay tuned for our next blog post in our migration series on custom processors. In this post, we will dive into what it takes to create a custom processor for a WordPress to Drupal migration.
Would you like help to create a more detailed plan for migrating your website to Drupal 8?
Contact Us – We would be happy to help!
About Drupal 8 Migrate
The migrate module has been moved into the core in Drupal 8, showing the community dedicated to making the process of upgrading between versions or migrating into Drupal easier. The migrate module takes advantage of the Drupal 8 Plugin system, offering developers with several plugin types that they can implement: MigrateProcessPlugin, MigrateSourcePlugin, and MigrateDestinationPlugin.
Quick Recap of our previous blog:
- Complete rewrite & moved into Drupal 8 core
- Out of the box support for D6 -> D8 and D7 -> D8
- Nodes, Users, Comments, Profiles, Taxonomies
- Content & Configuration
- Support for custom sources and destinations
- Processors for working with/manipulating data
Series Mapping:
Part 1: Migration Setup & Mappings
Part 2: Working with Processors
Part 3: Coming soon!
Drupal 8 Migrate Module Overview
The Drupal 8 migrate module that is shipped with core provides a set of API’s for setting up migrations. The module also provides extensible object-oriented base classes and interfaces for migration plugins including:
- Source & Destination Plugins
- Process Plugins
- Config Migration Mappings
While the migrate module has been moved into core, the contributed space still provides significant value and I wouldn’t recommend trying to build a migration without it:
- Migrate Plus: The Migrate Plus project provides extensions to core migration framework functionality as well as examples.
- Migrate Tools: The Migrate Tools module provides tools for running and managing Drupal 8 migrations.
Creating Migration Mappings
In Drupal 8 all of your migration mappings are done through configuration files. In Drupal 7 these migration mappings would have been done in classes through the $this->addMapping() function.
Configuration files provide the blueprint for the migrations, and there are two main types of configuration files that we will need to define:
- migrate_plus.migrate_group.<name>.yml
- migrate_plus.migrate.<name>.yml
Migration Group
The migration group is a configuration file and is similar to the idea of hook_migrate_api in Drupal 7. This configuration file defines a group of migration classes and configures global configuration/settings to be shared across the classes.
- id – Unique identifier
- shared_configuration – Defines shared configuration between all migration classes that are part of this group. **Example: setting the source database to use
- dependencies – Sets the dependencies for this set of migration classes to function
Let’s see an example:
Migration Configuration File
The migration configuration file is similar to a migration class in Drupal 7. At a high level, the migration config file defines the metadata and field mappings for a particular migration in Drupal 8. There are 5 key concepts that you need to be aware of in the migration configuration file:
- Definition – The definition of this migration class, its dependencies, and what migration_group it belongs to
- Source – What migration source should be used for this migration when it is run (i.e. where am I migrating my content from?)
- Destination – The destination for the migration (i.e. where am I migrating my content to?)
- Field Mappings – The mappings from Source -> Destination
- Processors – The processing of the source data so that it can be consumed by the destination
Let’s see an example:
Stay tuned for our next blog post in our migration series on custom processors. In this post, we will dive into what it takes to create a custom processor for a WordPress to Drupal migration.
Would you like help to create a more detailed plan for migrating your website to Drupal 8?
Contact Us – We would be happy to help!
As one of the top Drupal firms in the market, we get a lot of questions around Drupal 8 and its broad range of functionality, including Drupal 8 Batch Processing. We thought to start out the new year, we would offer our primer on Drupal 8 Batch Processing.
What is a batch job?
A batch job or batch processing is the execution of a series of jobs in a program on a computer without manual intervention (non-interactive). Strictly speaking, it is a processing mode: the execution of a series of programs each on a set or “batch” of inputs, rather than a single input.
In English, this means that it allows a computer program to break up a series of tasks into smaller chunks or pieces that run without any manual intervention to trigger.
When would I want to use this?
Drupal 8 Batch Processing jobs are valuable to use when there could be large amounts of data or long processes running that utilize a significant amount of memory. An example would be regenerating all URL aliases on your website. The “pathauto” module sets up a batch process when doing this to regenerate 25 aliases at a time, instead of trying to regenerate an entire site (think 5,000 – 500,000 entities) at one time that might cripple the system.
Why would I want to use this?
Performance & Scalability are the biggest reason to utilize Drupal 8 Batch Processing in your development. Batch jobs allow the processing of large amounts of data without relying on a single process to complete the task from start to finish in a single execution. This allows your server resources to be utilized in smaller chunks and freed up after each batch execution finishes.
Here are some questions you can ask when determining if you might need to create a batch process:
- Does the action I need to perform against these items have a per-item resource cost?
- If the action your performing requires loading or processing of each item individually, you should be looking to use batch processing to handle it. If you are performing a simple task, such as a bulk DB query that impacts all nodes in your database, it may not be required.
- Do I need to perform an action on a large number of entities?
- If the answer is yes, than you will likely gain significant performance benefits by utilizing batch processing to work through your task.
- Is there a finite set of data that I am performing actions on or can the dataset grow?
- If you are unsure about how big your data set will get, you should strongly consider batch processing. Not planning for this upfront could cause site downtime and lots of headaches later down the road.
- Even if your current data set is small, can it expand?
- For example, maybe your site only has 30 nodes at the moment, but that number will increase in the future. If this is the case, or you are building a module that you may want to contribute back to the community, you will likely want to look at batch processing as an option for handling this action.
How do I do this?
Creating a batch process in Drupal 8 is relatively straightforward. Here is what you will need to get started:
Demo_batch
- src/Controller/DemoBatchController.php
- demo_batch.info.yml
- demo_batch.routing.yml
- demo_batch.mybatch.inc
Demo_batch.routing.yml
The routing file defines a route, the Controller to be used, and the requirements to use it.
DemoBatchController.php
The controller tells Drupal what to do when the route (defined above) is accessed. In this case, we are creating a Batch Controller which will handle the processing of the batch job.
demo_batch.mybatch.inc
This includes file provides the callback functions for the controller to handle execution of the job. In this example, we are running through a migration task.