I run all my EC2 workloads on Amazon Linux. It comes with a superb AWS integration, a secure default configuration, regular security updates, and I can open AWS Support tickets if I run into any problems.
In late December 2017, AWS announced the successor of Amazon Linux: Amazon Linux 2.
AWS also announced that Amazon Linux 2018.03 is the last release for the current generation of Amazon Linux and will be supported until June 30, 2020. Therefore, you have to come up with a migration plan.
Amazon Linux 2 comes with the same benefits as Amazon Linux, but it adds some new capabilities:
- long-term support: Amazon Linux 2 supports each LTS release for five years
- on-premises support: virtual machine images for on-premises development and testing are available
- systemd: replacing SystemVinit
- extras library: provides up-to-date versions of software bundles such as nginx
Let’s dive into some of the changes in more detail. At the end of the post, I will also outline some pitfalls I encountered when migrating our Free Templates for AWS CloudFormation to Amazon Linux 2.
Further reading: Release Notes, FAQs, AWS Blog Post, Announcement
Long-term support
The Amazon Linux delivers a continuous flow of updates that allow you to roll from one version of the Amazon Linux AMI to the most recent. A yum update always moves your system to the latest Amazon Linux version. There were no versions of Amazon Linux available, only snapshots.
Amazon Linux 2 changes this. You will have Amazon Linux 2 versions that are supplied with updates for five years. Once a new Amazon Linux 2 LTS release becomes available, no breaking changes will be introduced by AWS for this release.
systemd
Amazon Linux uses SysVinit to bootstrap the Linux user space and to manage system processes after booting. This procedure is usually called init. One of the major drawbacks of SysVinit is that it starts tasks serially, waiting for each to finish loading before moving on to the next. This can result in long delays during boot.
Amazon Linux 2 uses systemd as the init system. systemd executes elements of its startup sequence in parallel, which is faster than the traditional serial approach from SysVinit. systemdcan also ensure that a service is running (e.g., it restarts a service if it crashed).
systemd is not just the name of the init system daemon but also refers to the entire software bundle around it, which includes:
journald: responsible for event logging (replaces syslog)udevd: device manager for the Linux kernel, which handles the /dev directory and all user space actions when adding/removing deviceslogind: manages user logins and seats in various ways.
I will not cover udevd and logind in this post. You should not get in touch with them as a normal user like me. Keep in mind that networking configuration is not controlled by networkd (also part of systemd software bundle). Instead, networking configuration is controlled by cloud-init which is triggered by systemd several times during boot. cloud-init handles early initialization of an EC2 instance (also works with other vendors).
Further reading: systemd man page
Reading logs from journald
To read all system logs (journal in journald terminology), starting with the oldest entry, run journalctl. The output is paged through less by default. Which means you can scroll down / up an entry with the DOWN / UP arrow keys, or scroll a full page down/up with the SPACE / b keys. Press the q key to quit. To reverse the order, run journalctl -r.
To show only the most recent journal entries, and continuously print new entries, run journalctl -f (like a tail -f).
There are many ways to filter the output. Based on priority, run journalctl -p err to get levels alert, crit, and err (using syslog log levels). Based on the unit, run journalctl -u sshd to get all entries for sshd. Check the further reading links for more information.
Keep in mind that some applications still write logs to /var/log. Journald also forwards logs to rsyslog which is configured (/etc/rsyslog.conf) to write some of them to files in /var/log.
Further reading: journalctl man page
Controlling systemd services
To start a service (unit in systemd terminology), you run:
systemctl start awslogsd.service
|
To make sure a service (unit in systemd terminology) is started during boot/reboot, you run:
systemctl enable awslogsd.service
|
There are many other commands. E.g., you can also reboot the system:
systemctl reboot
|
Further reading: systemctl man page
Extras Library
The Extras Library (aka Amazon Linux Extras Repository or Extras mechanism), provides a way to install up-to-date software bundles (topics in Amazon Linux 2 terminology) without impacting the stability of the rest of the operating system.
Extras Library is not covered by LTS!
To get a list of available topics, run:
$ amazon-linux-extras list 0 ansible2 available [ =2.4.2 ] 1 emacs available [ =25.3 ] 2 memcached1.5 available [ =1.5.1 ] 3 nginx1.12 available [ =1.12.2 ] 4 postgresql9.6 available [ =9.6.6 =9.6.8 ] 5 python3 available [ =3.6.2 ] 6 redis4.0 available [ =4.0.5 ] 7 R3.4 available [ =3.4.3 ] 8 rust1 available [ =1.22.1 =1.26.0 ] 9 vim available [ =8.0 ] 10 golang1.9 available [ =1.9.2 ] 11 ruby2.4 available [ =2.4.2 =2.4.4 ] 12 nano available [ =2.9.1 ] 13 php7.2 available [ =7.2.0 =7.2.4 =7.2.5 ] 14 lamp-mariadb10.2-php7.2 available [ =10.2.10_7.2.0 =10.2.10_7.2.4 =10.2.10_7.2.5 ] 15 libreoffice available [ =5.0.6.2_15 ] 16 gimp available [ =2.8.22 ] 17 docker=latest enabled [ =17.12.1 =18.03.1 ] 18 mate-desktop1.x available [ =1.19.0 =1.20.0 ] 19 GraphicsMagick1.3 available [ =1.3.29 ] 20 tomcat8.5 available [ =8.5.31 ] |
To install an topic, run amazon-linux-extras install <topic> (e.g., amazon-linux-extras install ruby2.4).
If you install (or only enable) a topic, a new repository (plus two for sources and debuginfo) is configured in /etc/yum.repos.d/amzn2-extras.repo.
Pitfalls
I migrated Free Templates for AWS CloudFormation to Amazon Linux 2. In the following, I will outline the problems I was faced with and how I worked around them.
The awslogs agent was renamed
The awslogs agent was renamed to awslogsd but you still install it via yum install awslogs.
You can start (activate in systemd terminology) awslogs with systemctl start awslogsd.service(shortcut: systemctl start awslogsd).
The awslogs agent does not support journald
awslogs agent cannot read logs directly from the journal. journald fowards all logs to rsyslogwhich is configured (/etc/rsyslog.conf) to write some of the logs to files in /var/log from where the awslogs agent can pick them up.
Where are the log files?
/var/log does not contain all system logs anymore.
If in doubt, you can access all system logs with journalctl.
Ruby is missing
Ruby is no longer installed by default. This breaks cfn-init if you want to install RubyGems.
You can install Ruby 2.0 with yum install ruby or Ruby 2.4 with amazon-linux-extras install ruby2.4.
netcat is missing
netcat (or nc) is no longer installed by default.
You can install ncat with yum install nmap-ncat, but this will install nmap based ncat which behaves differently (e.g., no -z flag anymore). Learn more
Nginx package not available by default
nginx is no longer part of the default repository.
$ yum install nginx Failed to set locale, defaulting to C Loaded plugins: langpacks, update-motd No package nginx available. Error: Nothing to do |
To install nginx, use the new Amazon Linux Extras Repository amazon-linux-extras install nginx1.12.
EPEL repository is missing
The EPEL repository (Extra Packages for Enterprise Linux) is no longer installed by default or available to install. The Extras Library replaces the EPEL repository but contains only a fraction of the packages which may causes troubles during your migration.
NAT and ECS optimized AMIs are missing
NAT and ECS optimized AMI are not available. You can replace your NAT instances with NAT Gateways to get around this problem. But for ECS workloads there is no easy workaround. I advise waiting for news from AWS regarding the ECS optimized AMI.
cfn-init is not integrated with the Extras Library
You can not install packages from the Extras Library with the package mechanism in cfn-initeasily. cfn-init is the way how you can install software onto EC2 instances managed by CloudFormation.
There can either run amazon-linux-extras enable <topic> before running cfn-init which than can install the package by using the package mechanism. Or you can use two config sets. The first config sets uses the command mechanism to enable the topic. The second config set uses the package mechanism to install the enabled package. You have to use two config sets because commands run after package installation. Here is an example:
AutoScalingGroup: Type: 'AWS::AutoScaling::AutoScalingGroup' Properties: # [...] LaunchConfiguration: Type: 'AWS::AutoScaling::LaunchConfiguration' Metadata: 'AWS::CloudFormation::Init': configSets: default: [extras, config] extras: commands: a_enable_nginx: command: 'amazon-linux-extras enable nginx1.12' test: "[ ! grep -Fxq '[amzn2extra-nginx1.12]' /etc/yum.repos.d/amzn2-extras.repo ]" config: packages: yum: nginx: [] # will install nginx1.12 Properties: # [...] UserData: 'Fn::Base64': !Sub | #!/bin/bash -x /opt/aws/bin/cfn-init -v --stack ${AWS::StackName} --resource LaunchConfiguration --region ${AWS::Region} /opt/aws/bin/cfn-signal -e $? --stack ${AWS::StackName} --resource AutoScalingGroup --region ${AWS::Region} |
Summary
Amazon Linux 2 is the new default for running Linux workloads on AWS. Amazon Linux 2 benefits from systemd, LTS, and a new extras library. There are a few pain points when migrating, most notably the missing EPEL repository. Besides that, you should spend some time to understand how systemd works, because that’s central in modern Linux operating systems.
It’s time to plan your migration from Amazon Linux now!