Deploying a Private Puppet Forge

08 Oct 2018

If your using Puppet or Puppet Enterprise with a Control Repository (which you definitely should be), then you may have noticed that the instructions mention two main ways of getting this working:

Use the Puppet Forge directly
Mirror modules to a local git server and use from there.

Many larger organisations don’t like to download Puppet Modules (or any code!) directly from the Internet, since this means:

Upstream code hacks might be automatically downloaded
Business continuity issues if repositories go offline temporarily or permanently

For Puppet, the prevailing wisdom to deal with this has been to host all forge modules in their own git repositories. This addresses the business continuity problem and somewhat protects against poisoned upstream repositories but there are a few problems with this approach:

It’s a lot of work to keep your local copy up-to-date
If you have lots of forge modules, you will have lots of local git copies
Some users may be tempted to edit code in the local git repositories instead of contributing to the upstream project, resulting in a diverged module that can no longer be easily upgraded
Depending how your git import/update process works, you may be worse off security-wise then if you had just used the Puppet Forge. For example:
- Tags can be moved or deleted
- Branches can be force pushed or deleted

The vast majority of Puppet Forge users just want:

Puppet Forge modules hosted locally for faster downloads and more reliable deployment
The package tarball! - Since Puppet modules are not compiled before release, we have everything we need to audit the module in the build artifact.
No messing around with git
Install modules from local repository exactly the same way as from the Puppet Forge

What software is needed to host a private Puppet Forge?

The main Puppet Forge website doesn’t just host packages as a bunch of tarballs in a directory structure. It implements the custom Puppet Forge REST API.

REST itself being a way to build self-describing web interfaces.

Because of the need to implement an API, you can’t just host files on a plain webserver, you need something called a repository server to make the files and the API available.

In this article, we’ve evaluated the current crop of repository servers for you so that you can be up and running as fast as possible.

Note that we are specifically talking about hosting The Puppet Forge itself in this article. If you just want to snag some binaries using Puppet you can totally do that… today!

Nexus (Pro/OSS)

SonaType Nexus does not have Puppet Forge support in any version:

“We do not currently have puppet forge support in Nexus Repo” - June 2018

Perhaps it will be added at some point in the future…

Pulp

We evaluated Pulp in May 2018, since then there have been some more commits, so these issues may since have been resolved

Pulp works well if your using RedHat Satellite but this mode of operation relies on Satellite itself pushing modules to the built-in Puppet Master. This won’t work nicely with Puppet Enterprise and R10K/Code Manager.

At time of evaluation, we were unable to get Pulp working as a Puppet Forge compatible server due to missing /modules endpoint.

There are some tickets around Puppet Forge support:

1848 since 2 yrs
1536 closed 2 yrs ago, support was presumably broken by an API change

Artifactory

JFrog Artifactory was the stand-out repository of the crowd. Not only does it offer full support for the Puppet Forge API, it even offers a click-through GUI to set it up which only takes about 10 minutes and that’s if you take it slow and read all the instructions.

If your in a corporate environment and can afford the licence, using Artifactory to host your local Puppet Forge repository is a no-brainer.

unibet/puppet-forge-server

Kudos to Unibet for making and releasing their forge_server project as Free Software!

The server itself looks good and works great. If you just want a local cache of forge modules with no fuss and no frills, then this might be the repository server for you.

If your in a corporate environment there are a couple of big gotchas which you would have to work around yourself to put this into production:

Unsecured UI
Unsecured uploads

Set-up

Assuming your in a corporate environment and have chosen Artifactory, the steps to setup are really simple. In-fact, their only shown here to illustrate just how simple it is:

Create the Puppet repository

Create a new repo

Create a new repository

Choose puppet

Choose Puppet

Finish

That’s it(!)

Find the repository URL

Click setmeup

Click Set Me Up

Get the URL

Grab the URL

Configuring Puppet Enterprise

With your repository URL handy, your ready to configure Puppet Enterprise to use your Private Forge instead of the Internet.

Puppet Enterprise is configured by setting the forge_settings parameter of the puppet_enterprise::master::code_manager class. The easiest way to do this is with Hiera, like this:

CONTROL_REPO/data/common.yaml

puppet_enterprise::master::code_manager::forge_settings:
  baseurl: 'https://repo.megacorp.com/artifactory/api/puppet/puppet'

Chicken and Egg problem

If you look closely at the above, you’ll notice we have a Chicken and Egg problem:

Our Puppetfile contains forge modules which cannot be deployed until we have configured Puppet Enterprise to use our Private Forge
Configuring the Private Forge relies on Hiera data in our Control Repository, but we can’t deploy it because of missing modules.

At Declarative Systems, our attitude is: “Screw the chicken and the Egg! You come first”.

We can workaround this problem like this:

Manual fix: Login to the Puppet Enterprise console and manually classify with the connection details for your Private Forge
Edit the global Hiera configuration to insert an additional layer and drop a .yaml file on the Puppet Master to configure the Private Forge. This plays nicely with automated Puppet Master deployments and/or DR procedures. If your caring for your Puppet Master(s) as pets your doing it wrong.

Manual Fix

Find where class puppet_enterprise::master::code_manager in the Node Classifier and set the value for the forge_settings parameter to be a hash which defines a key baseurl containing the URL you obtained from Artifactory.

Global Hiera configuration

/etc/puppetlabs/puppet/hiera.yaml

# hiera.yaml
#
# This file has been modified from the vendor's copy:
#   1. include this notice
#   2. read from /etc/puppetlabs/puppet_enterprise.yaml as a top-level hierarchy
---
# Hiera 5 Global configuration file

version: 5

# defaults:
#   data_hash: yaml_data
# hierarchy:
#  - name: Common
#    data_hash: yaml_data
hierarchy:
 - name: Classifier Configuration Data
   data_hash: classifier_data
 - name: 'Puppet Enterprise Settings'
   data_hash: yaml_data
   path: /etc/puppetlabs/puppet_enterprise.yaml

/etc/puppetlabs/puppet_enterprise.yaml

puppet_enterprise::master::code_manager::forge_settings:
  baseurl: 'https://repo.megacorp.com/artifactory/api/puppet/puppet'

Adjust the value for baseurl with the URL from Artifactory

Activating configuration

The settings changes won’t take effect until Puppet has been run on the Master:

puppet agent -t

After this run completes (you must already be using Code Manager), Puppet Enterprise will be configured to use your Private Forge.

Deploying code

To deploy code using your new Private Forge, just run the puppet code deploy command as usual and Code Manager will use your new Artifactory server.

You can look at the Artifactory logs to confirm correct operation. If there’s nothing in the logs and you get strange errors from Puppet Enterprise, the very first thing to do is make sure your Artifactory server is reachable from the Puppet Master:

curl https://repo.megacorp.com/artifactory/api/puppet/puppet

Loading modules

You can choose to configure Artifactory as a smart proxy, where modules requested are cached locally. If this is the case, you have no more work to do since Artifactory will allow the use of any version of any Puppet Forge module in this mode.

If you have more stringent security requirements, you probably configured Artifactory to only serve approved modules.

In this case, there are a couple of ways to load Puppet Modules into Artifactory:

Download manually from Puppet Forge, Upload manually using Artifactory Web UI
Script the process somehow (both Puppet Forge and Artifactory have full REST APIs)

Since we’re all about Automation, we already developed a simple Ruby script to:

Read the modules described in your Puppetfile
Download them to a directory
Upload the entire directory back to Artifactory

The download and upload steps are separate in-case you need to connect to different network for different parts of the job. Separate jobs also allows you to manually inspect the contents of each Puppet Module tarball before you publish to Artifactory.

This experimental script uses basic authentication to do the uploads for the moment. We’re keen to hear your feedback on it.

pushmepullme

Pro Tip

Use Artifactory to serve all the binaries you need, and configure Puppet Enterprise to download from it.

Gotcha: Error 400

If using a private forge (Artifactory) to deploy puppet code using puppet code deploy user will encounter Error 400 when:

The module you are trying to download exists in the private forge
..But the exact module version you are requesting does not

See ENTERPRISE-1187 for More info

geoffwilliams@home:~$