Content and Tooling – Posts

Blog: A one stop shop for everything Puppet.Dsc!

Tue, 27 Feb 2024 00:00:00 +0000

A brief history

It has taken a couple of iterations to get to where we are today with the integration between Puppet and Powershell’s DSC. We’ve taken a number of approaches over recent years, and admittedly it hasn’t always been the smoothest of roads. The idea for this integration between two configuration management tools stem from their overwhelming similiarties, DSC, as described by Microsoft themselves is “a management platform in PowerShell that enables you to manage your IT and development infrastructure with configuration as code.”, so essentially the end goal of both Puppet and DSC is the same. We aim to empower users to manage and configure their IT infastructure at scale efficiently.

Naturally, this got the brains at Puppet here thinking about how we can leverage the power of DSC, through the even greater power of Puppet. And so the saga began!

puppetlabs-dsc

First off, puppetlabs-dsc. This module would generate Puppet types based on DSC Resources MOF (Managed Object Format) schema files. puppetlabs-dsc was an innovative idea, and essentially generated puppet types on the fly, which all ran through a generic powershell provider. Essentially, when you ran puppetlabs-dsc it would create a Puppet Type for each resource defined in your manifest, all of which would utilise the powershell provider. Each time Puppet was ran, the provider would generate the equivalent DSC powershell script, which contained the Invoke-DscResource cmdlet, which would invoke DSC as if you were running it directly.

WindowsFeature IIS {
Ensure = 'present'
Name = 'Web-Server'
}

…would translate to this in Puppet:

dsc_windowsfeature {'IIS':
dsc_ensure => 'present',
dsc_name => 'Web-Server',
}

So what happened? As time went on the module became fragile and stale and required regular manual maintenance.

puppetlabs-dsc_lite

Our next stop on this trip down memory lane is with the puppetlabs-dsc_lite module (for simplicities stake, lets call this dsc_lite). Dsc_lite was like puppetlabs-dsc, but minus any guardrails and Puppet’s native data types to hold your hand. Like its predecessor, it also created puppet types for each dsc resource defined in your puppet manfiest, whilst executing the equivalent dsc powershell scripts through a generic powershell provider. This was a much more lightweight, maintainable and was a “one size fits all” solution, unlike it’s puppetlabs-dsc counterpart.

There were some notable differences between how you would define a manifest in puppetlabs-dsc vs dsc_lite. The below manifest in puppetlabs-dsc

dsc_windowsfeature {'IIS':
dsc_ensure => 'present',
dsc_name => 'Web-Server',
}

would have an equivalent manifest in dsc_lite like

dsc {'iis':
resource_name => 'WindowsFeature', # name of DSC resource
module => 'PSDscResources', # name of DSC module on powershell gallery
properties => { # name and value of properties to be passed to Invoke-DscResource
ensure => 'present',
name => 'Web-Server',
}
}

However, this did have its drawbacks!

Required extensive knowledge of native dsc
puppetlabs-dsc_lite has no way of knowing whether the values for each property you provided were correct until after dsc invocation. In other words, no way to tell if you goofed til run time.

So, back to the drawing board we went..

Puppet.Dsc

And ultimately, due to all to the reasons above, we were finally led to where we are today, Puppet’s auto-generated DSC modules. Thanks to the amazing wizardy that is Puppet.Dsc, we are able to take any DSC module directly from the Powershell Gallery (DSC’s answer to Puppet forge), and generate an equivalent Puppet module. Puppet.Dsc would retrieve a module from the gallery and convert each DSC resource into Puppet Resource API types and providers. This meant that user’s could manage their Puppet DSC modules, the same way they would with their regular old Puppet modules, just installing what module and version you needed in your environment, no fuss.

This was and has been the recommended way to run DSC through Puppet in recent years.

Like the above, there are some limitations to this method. Not all DSC modules are compatible with Puppet.Dsc.

Not all DSC and puppet properties have a one-to-one mapping. I.e. Dsc keys and Puppet namevars may appear the same on the surface, but behave differently when passed solely in a manifest.

Such discrepancies can arise and cause unexpected behaviour. Luckily, these instances are rare, and usually very niche.

Under the hood all three approaches are the same, all they are is a DSC abstration, ran through the familiar interface of a Puppet manifest.

Getting started with Puppet.Dsc

The Puppet.Dsc PowerShell module is available on the PowerShell Gallery; If you just want to get your hands on the PowerShell module to read the help and experiment on your own, you can install it like any other PowerShell module:

Install-Module -Name Puppet.Dsc
Import-Module -Name Puppet.Dsc
Get-Command -Module Puppet.Dsc
Get-Help -Name New-PuppetDscModule -ShowWindow

Puppetizing a PowerShell module

So, you want to turn a PowerShell module with DSC Resources into a Puppet module? You can do this with a single command:

# If you run this command without the PassThru flag it returns nothing to the output stream.
# Any conversion errors will be in the error stream, as usual.
# If you'd like a deeper look at what it's doing, try running it with the Verbose flag.
New-PuppetDscModule -PowerShellModuleName ComputerManagementDsc -PassThru

Unfortunately, the output from running it isn’t all that exciting:

Behind the scenes, the command:

Uses the PDK to create a new Puppet module
Searches the PowerShell Gallery for the ComputerManagementDsc module and vendor it into the new Puppet module
Converts each of the DSC Resources from the vendored PowerShell module into a Puppet Resource API type and provider
Updates the Puppet module’s metadata and documentation based on the PowerShell module.
Outputs the generated module to $PWD.Path/imports/YourDscModule

At this time, for best results we strongly recommend you run the command with administrator privileges in a Windows PowerShell 5.1 session. While it does work without administrator privileges, the function cannot fully map nested CIM instances without administrator privileges. For more information on this process, check out the about_Puppetization HelpFile via Get-Help for a more in-depth documentation of this process.

Using puppetized modules

Prerequisite: Ensure the module is installed within your environment’s modulepath, or path to the directory containing the module is supplied to your puppet apply command using the --modulepath=path/to/modules/dir flag. See here for more information on modulepath usage.

On the one hand, there’s not much to say about using the Puppetized modules with DSC Resources; you use them precisely like any other Puppet module, after all. On the other hand, there’s some strong utility worth going over in more detail you may or may not be aware of!

To get the most value from your workflow, we strongly recommend you author your manifests that include Puppetized DSC Resources in VSCode with the Puppet extension enabled.

When we open a file named example.pp to edit, the Puppet extension automatically begins to load; once this is finished, we have access to a ton of useful authoring support, including intellisense.

For example, if we’ve installed the computermanagementdsc Puppet module, we need only type dsc_execution before IntelliSense kicks in and helps us out:

Note that it tells us not just what available resources might match, it also displays the available documentation for each resource.

We also get IntelliSense for each property we want to pass, too; again, this doesn’t just show the available options but also includes any available documentation for each property.

You can also review the documentation for a property by hovering over it:

Where this combination really begins to sing is when you’ve begun authoring a manifest and something goes wrong.

The image above shows that the file name in the left-hand pane has changed colors and is now red. It also underscores two entries in the manifest that VSCode has highlighted - one with orange underlining, indicating a warning, and one with red underlining, indicating an error. Finally, down in the bottom left corner it highlights the problem symbols for errors and warnings, both of which display a count of one.

VSCode (via the extension) knows something is wrong with our manifest; we can click on those symbols to bring up the Problems Pane and investigate:

This tells us there’s a warning about the indentation of a hash rocket and some sort of syntax error preventing the extension from parsing line 3.

A quick look at the code tells us that we’re missing a comma in the resource declaration on line 2, and as soon as we add that comma in we get confirmation:

All that remains now is the formatting issue. Luckily, VSCode is able to automatically handle this for us via the Format Document command:

Which we can see does in fact move the hash rocket and resolve our last problem:

We’re ready to apply this manifest!

puppet apply ./example.pp

Only… there was a problem; without reading the documentation for dsc_executionpolicyscope thoroughly enough, we specified Machine instead of MachinePolicy!

Puppet gives us back a pretty good error message though:

Error: Parameter dsc_executionpolicyscope failed on Dsc_powershellexecutionpolicy[[OPS-213]]: dsc_powershellexecutionpolicy.dsc_executionpolicyscope expects a match for Enum['CurrentUser', 'LocalMachine', 'MachinePolicy', 'Process', 'UserPolicy'], got 'Machine'

We get an explicit list of the allowed values before it tries to invoke DSC with the Set action. This is unlike our prior implementation in puppetlabs-dsc_lite, which has no way of knowing whether the values you’ve provided are correct.

Once we correct our manifest to specify LocalMachine and re-run the command, we get something else:

Notice: Compiled catalog for michael.lombardi-pf12aa84 in environment workstation_production in 1.04 seconds
Notice: /Stage[main]/Main/Dsc_powershellexecutionpolicy[[OPS-213]]/dsc_executionpolicy: dsc_executionpolicy changed 'Unrestricted' to 'RemoteSigned'
Notice: dsc_powershellexecutionpolicy[{:name=>"[OPS-213]", :dsc_executionpolicyscope=>"LocalMachine"}]: Updating: Finished in 0.809555 seconds
Notice: Applied catalog in 1.68 seconds

This highlights a change from both of our prior implementations: you now get property-by-property reporting for DSC Resources when using Puppet! This implementation knows not just that a DSC Resource was out of sync and therefore applied, but it can tell you exactly which properties were out of sync, what their value was, and what it has been set to in this run.

This brings DSC Resources in line with all of the rest of the resources you manage with Puppet and drastically improves the usefulness of your run reports!

Some helpful resources

You’re all set!

You now know how to start generating your own puppet DSC modules. Happy puppeting! :-)

Blog: Firewall Update - Converted to the resource_api

Fri, 25 Aug 2023 00:00:00 +0000

Firewall Update - Converting to the resource_api

What has changed and what will I need to update?

While there have been significant changes to the module, when it comes to the users experiance, there are only six that I would deem backwards incompatible:

The provider attribute for the Firewall type, which sets whether the rule is for iptables or ip6tables, has been renamed to protocol. This is for two reasons; firstly due to the resource_api specifically forbidding the use of provider as an attribute name as it is deemed bad practice and secondly to bring it in line with its Firewallchain counterpart.
- Of note, the attribute has also been updated to allow IPv4 or IPv6 to be passed in place of iptables or ip6tables as it is with Firewallchain, though either style is accepted as valid.
The action attribute has been removed as it was merely a restricted version of the jump attribute, with both of them managing the same function. Though this was reasoned as a way to enforce the use of generic parameters I found it to be needlesly complex and so jump should now be used in all situations.
Strict types have been declared for all attributes, the type of information that can be passed to each attribute is now validated on a higher level.
The port attribute has been removed as it has been deprecated for several years now and is no longer relevant.
Attributes that allow both arrays and negated values have now been updated.
- For attributes that require that all passed values be negated as one, you now merely have to negate the first value within the array, rather than all of them, though negating all is still accepted.
- For attributes that allow passed values to be negated seperately this is not the case, with each value being negated, or not negated as it were, seperately. All attributes in this situation are noted within their personal description.
One last change, that while not backwards incompatible, I feel shoul be mentioned is that the sport and dport attributes have now been updated to allow ranges of values to be passed with the : symbol as a seperator, as this is how they are passed into the iptables command. The original form of the seperator -, is still accepted however.

Overall, while there may be some changes that you will have to make to you manifests in order to make them compliant with the new form of the module, I have done my best in order to keep them to a minimum and feel that, for the most part, I have succeeded.

Example manifest comparison

Some example manifest changes, pulled from the tests, are show below.

An manifest applying a hoplimit on ip6tables:

firewall { '571 - hop_limit':
ensure => present,
proto => 'tcp',
dport => '571',
action => 'accept',
hop_limit => '5',
provider => 'ip6tables',
}

firewall { '571 - hop_limit':
ensure => present,
proto => 'tcp',
dport => '571',
jump => 'accept',
hop_limit => '5',
protocol => 'IPv6',
}

A manifest blocking access to a pair of ports on iptables:

firewall { '560 - negated ports':
proto => `tcp`,
sport => ['! 560-570','! 580'],
action => `accept`,
}

firewall { '560 - negated ports':
proto => `tcp`,
sport => '! 560:570','580',
jump => `accept`,
}

Main improvements

In regards to how these changes will improve the modules usage, I will be honest and say that they will mainly effect the maintanence and future improvement of the module, increasing the ease with which it can be debugged and with which requested fixes and improvements can be made. While there is some functional improvement, with the module now more efficiently parsing the returned/given values, it has been largely limited by the necessity of each change in rule being applied with it’s own command.

Future improvements

While this has been a large improvement to the module there are many more that can be made for the future:

A community suggestion was made to implement the optparse ruby module in parsing the retrieved and applied rules from their correct form to the Puppet hash values that we set in or manifests. This could help to replace the most complex part of the updated module and reduce the overall amount of code within the providers and further simplify the module as a whole.
Many of the currently implemented attributes can be negated but do not currently allow it, this is especially true of the boolean values, the code for which will need to be updated specifically to allow this.

Closing Thoughts

Implementing the resource_api into this module is something that I have long considered a vital improvement and one that I am happy that I was able to close of myself.

I have learned alot in the process and have thouroughly enjoyed the work with my hope being that everyone who utilises or works on the module going forward is appreciative of the changes made and find the module as a whole much easier to work with.

David

Blog: Deprecation of puppetlabs-stdlib functions

Fri, 12 May 2023 00:00:00 +0000

Throughout the massive push to get our modules Puppet 8 compatible, we have also been working on finally removing old deprecated functions that have been living in the codebase for quite some time. Keep an eye out for the removals as this may impact your code base if you are using them.

Impacted Functions

abs
camelcase
capitalize
ceiling
chomp
chop
dig44
downcase
getvar
hash
has_key
is_array
is_bool
is_float
is_ip_address
is_ipv4_address
is_ipv6_address
is_numeric
is_string
lstrip
max
min
private
round
rstrip
sort
sprintf_hash
strip
upcase
unique
validate_absolute_path
validate_array
validate_bool
validate_hash
validate_integer
validate_ip_address
validate_ipv4_address
validate_ipv6_address
validate_numeric

This list is not exhaustive and for more information please check out the module source code here.

Thank you to everyone that contributed and helped us!

I would like to take the opportunity to do a massive shout to the community with helping us on this one, it has been a massive team effort. None of this would have been possible without the dedication from everyone and in particular smortex, b4ldr, bastelfreak, alexjfisher and MartyEwings.

Need to chat to us?

Catch us on Slack and during office hours weekly (Tuesday 2pm BST).

Blog: puppet-lint: Already initialized constant warnings

Thu, 16 Mar 2023 00:00:00 +0000

For a while now, the voxpupuli-puppet-lint-plugins meta gem has provided the community with a number of amazing plugins for puppet-lint.

With the latest release of puppet-lint and and Vox’s plugins gem, a few things have changed that might cause a bit of friction when updating.

I wanted to take a second to go over a few scenarios that might help you on your journey.

Problem

First of all, lets talk about the problem.

In puppet-lint v3.1.0 we adopted two plugins which are now included as default checks:

Prior to v4.0.0 of voxpupuli-puppet-lint-plugins, these plugins were included in the meta gem.

So now we have a scenario where it’s possible to update puppet-lint (>= 3.1.0) and end up with duplicate plugins installed.

In this case, users may see warnings like this:

warning: already initialized constant PuppetLint::CheckLegacyFacts
warning: previous definition of CheckLegacyFacts was here
warning: already initialized constant LEGACY_FACTS_VAR_TYPES
....
warning: already initialized constant TOP_SCOPE_FACTS_VAR_TYPES
warning: previous definition of TOP_SCOPE_FACTS_VAR_TYPES was here

Not the end of the world, but it’s extra noise that you don’t want in your pipeline.

Help! This is happening to me!

So you are seeing these warnings? Read on…

What we are seeing here is a soft dependency conflict.

We have two identical plugins registering the same code inside puppet lint. That’s what is causing the already initialized constant warning.

The best way forward is to clean up our environment and start again.

Remove voxpupuli-puppet-lint-plugins and it’s dependencies

We can start by removing dependencies of voxpupuli-puppet-lint-plugins.

The command below will do just that. We grab the dependencies of Vox’s meta gem and remove them one by one.

gem dependency voxpupuli-puppet-lint-plugins --pipe | grep "puppet-lint" | cut -d" " -f1 | xargs gem uninstall -aIx

Note: Take extra care when running commands like this! 👀

After the previous command has completed, we should be left with the voxpupuli-puppet-lint-plugins meta gem.

Let’s remove it with the following command:

gem uninstall voxpupuli-puppet-lint-plugins

..and check to see if everything has really gone.

gem list | grep "puppet-lint"

Tldr; It should return nothing! Hurrah 🎉

Re-install the latest gems

Now we are free to re-install puppet-lint and voxpupuli-puppet-lint-plugins

gem install puppet-lint
gem install voxpupuli-puppet-lint-plugins

Once they have installed you can enjoy a warning free puppet-lint experience with all of the joys from voxpupuli-puppet-lint-plugins!

… unless you have linting issues!

Blog: Removing puppet-modules-gems

Tue, 10 Jan 2023 00:00:00 +0000

What are `puppet-module-gems` and why are they now considered harmful?

Okay, that might be a bit of a melodramatic take on the situation, but the puppet-module-gems really did serve a useful purpose at one point. Let’s go back over why it exists and what’s changed in the meantime.

We’ll start with Bundler–a local environment manager for Ruby projects. It runs a Ruby project with all of its required dependencies in a sandbox, meaning that you don’t need to install gems globally. This is great for repeatability and means that each project can have the specific gem versions it requires without interfering with other projects that might need different versions.

This worked great for Puppet back in the good old days when things were simple. All the gems Puppet used were specified in the Gemfile and Bundler environments just worked. But then the Ruby ecosystem evolved and gems started taking advantage of new functionality. Version 1.2 of a gem might run on Ruby 1.8, but an upgrade to version 1.4 was required to use Ruby 2.0 and then Ruby 2.5 needed yet another version. This got to be rather complex, and worse Bundler couldn’t resolve these dependencies properly! We had to do it ourselves, making the Gemfiles brittle and harder to read and maintain.

CI was breaking, module authors were struggling to test on multiple platforms, users were having a hard time validating updates. Clearly we needed a solution, and the puppet-module-gems were the answer. We created meta-gems for each Ruby version we supported and specified the proper dependencies in these gems. Then the Gemfile would conditionally depend on the properly versioned meta-gem based on the version of Ruby running.

This worked reasonably well, but had its challenges. The dependencies were opaque and the extra level of abstraction meant that looking at the Gemfile didn’t really tell you much about what gems would be installed or used. Worse, we had to manually resolve and specify all the dependencies for every Ruby version, and keeping up with those turned out to be easier said than done. We often fell behind and our more adventurous users would regularly remind us of that. The situation certainly wasn’t ideal, but it was a workable compromise.

Until Bundler got smarter, learned how to resolve Ruby version dependencies, and made this workaround no longer necessary.

What’s Changing

With these Bundler updates, we no longer require such tight control over dependencies; Bundler will do the right thing and resolve them properly, taking gems' supported Ruby versions into account. We have decided to stop maintaining the puppet-module-gems and instead move fully to managing gem dependencies with each module’s Gemfile. This is standard practice for the Ruby ecosystem, so it will be easier to maintain in the long term. Since it’s more flexible, it will also better allow downstream users to modify according to their own needs.

This is especially relevant now as it means that the puppet-modules-gems have not been updated to account for Ruby 3, which the upcoming Puppet 8 release will release with. This means that any module that has not yet converted to the new format will be not be testable on Puppet 8.

What You Need To Do

If you maintain your own Puppet modules, the rest of this post is for you. If you only use modules from the Forge, then no action is necessary on your part, other than to file bug reports with a link to this blog post if you notice modules which should be updated.

Thankfully, the solution to adopting the new standard is relatively simple, at least for those who are using the PDK and the default templates. We’ve already made the updates for you and you’ll just need to update your module with these changes. This will replace puppet-module-gems in the Gemfile with the actual dependencies that the meta-gem represented.

$ pdk update

An example of the changes this update includes can be seen on many of our modules, with one example shown here.

In the event that you are not using the default PDK templates then you will need to update each Gemfile yourself. Look at the config default values here and copy what you need into your own template repository, updating them according to your own needs.

If you are not using the PDK at all, you can find an example of a current Gemfile below, which you can copy and modify as you need. Tracking the default versions listed under the Gemfile key in config_defaults.yml will help you keep these versions correct as you update over time.

group :development do
# These where already set in pdk templates and so may not be relevant to you
gem "json", '~> 2.0', require: false
gem "voxpupuli-puppet-lint-plugins", '~> 3.0', require: false
# The main group of rules we are adding
gem "facterdb", '~> 1.18', require: false
gem "metadata-json-lint", '>= 2.0.2', '< 4.0.0', require: false
gem "puppetlabs_spec_helper", '>= 2.9.0', '< 4.0.0', require: false
gem "rspec-puppet-facts", '>= 1.10.0', '< 3.0.0', require: false
gem "codecov", '~> 0.2', require: false
gem "dependency_checker", '~> 0.2', require: false
gem "parallel_tests", '~> 3.4', require: false
gem "pry", '~> 0.10', require: false
gem "simplecov-console", '~> 0.5', require: false
gem "puppet-debugger", '~> 1.0', require: false
## We pin these in order to keep the style rules static
gem "rubocop", '= 1.6.1', require: false
gem "rubocop-performance", '= 1.9.1', require: false
gem "rubocop-rspec", '= 2.0.1', require: false
gem "rb-readline", '= 0.5.5', require: false, platforms: [:mswin, :mingw, :x64_mingw]
# Added locally and used by us as part of the release process, may not be relevant to you
gem "github_changelog_generator", require: false
end
group :system_tests do
gem "puppet_litmus", '< 1.0.0', require: false, platforms: [:ruby]
gem "serverspec", '~> 2.41', require: false
end

Conclusions

Since we are not un-publishing the existing puppet-module-gems meta-gems, we don’t expect many people to be impacted by this change. Many module authors won’t even notice it, as they’ll pdk update their modules naturally at some point and inherit the updates before it becomes critical for them. What this does mean, though, is that you should get more used to running pdk update regularly as this is the standard way in which updated dependencies and support for new Ruby versions will be delivered.

Good luck, and happy Puppeting!

Blog: Puppet Approved Modules are back!

Fri, 28 Oct 2022 00:00:00 +0000

What does Approved mean?

An approved module means we have checked against a specific criterion, if the module meets the suggested criteria, we do not officially offer support, however it is a module that we would recommend using. For a more extensive write-up on exactly what this means check out our definition on the Puppet Forge.

What does it look like?

When your module is ‘Approved’ it will receive an ‘Approved’ badge on the Puppet Forge. Here is an example of what an ‘Approved’ module will look like:

How can I get my module ‘Approved’?

If you feel that your module is actively maintained and meets the guidelines published here then all you need to do is create an issue in puppet-approved-modules and one of our team will get around to checking it as soon as they can. They will provide feedback on anything they need to, or award you an ‘Approved’ badge (Yes, it is that simple!).

Catch us on Slack for any further information or questions.

Blog: Changes to Supported Modules

Thu, 30 Jun 2022 00:00:00 +0000

We are continually reviewing our Supported modules list and understanding usage and value to users. We want to ensure that the CAT team is prioritizing support for the modules that matter the most to you.

We are taking the step to remove Puppet support from the below listed models, this change will happen effective 15 July 2022. While these will not be supported any longer by Puppet, the repos will stay active for use and community contributions, as such they are not going away.

Why are we making these changes?

We are making this move in order for us to focus on providing more robust and quality content for our other existing modules and to also allow us to bring to our user base new and exciting content that can help you manage and run your environments.

Reminder on reporting of Issues

In order to better triage and prioritize work surrounding each module, please remember to raise issues to the GitHub repositories. We have deactivated new submissions for the Modules Jira Project and have moved existing tickets into our workflow to be reviewed.

Catch us on Slack for any further info or questions and look out for some exciting updates in the future.

Blog: Community Day returns on the 16th of May! 🎉

Wed, 11 May 2022 00:00:00 +0000

We are excited to announce that community day is returning on May 16th 2022 and will be continuing every Monday for the foreseeable future.

Why is community day so good anyway?

It means we get to spend time with our amazing community, understand how our content is being used and help level up our modules and tools by addressing issues and pull requests.

Community day benefits us as a team too!

Promotes collaboration and healthy team work ❤️
Upskills new and existing engineers 👷🏾‍♀️
Increases productivity 💯
Overall increase of team morale 👍

What to expect

We are here to help you!

🕘 We will be available between 9am and 5pm UK time
🏢 We hang out on GitHub and our community slack channel
📚 We want to help you with issues, pull requests and product related questions

Signing off

That’s it for now! We are excited to see you all on Monday.

For more information about who we are and what we do in the Content and Tooling team, check out our intro blog post.

Blog: Puppet Runtime Manager 0.2.0 Release 🤘

Fri, 06 May 2022 00:00:00 +0000

We are happy to announce that version 0.2.0 has been released. 🎉

With this release we introduce the build and validate commands along with a number of other fixes.

Build 🏗

After developing a tool you generally want to share it with the wider community or your team.

Let’s take a quick look at what that might look like.

prm build provides functionality that will package the current directory and ensure that the resulting artefact is a valid PRM tool.

A basic project should contain at least a prm-config.yml and a content directory.

Assuming that your project structure looks OK, you can run prm build.

The resulting archive will be created in the pkg directory by default. However you can change the location that the package is created in by providing --targetdir flag.

Your package can now be installed by anyone with prm install!

Validate ✅

A validator is a tool which can read and validate a codebase of Puppet content.

prm validate enables a content creator to quickly validate Puppet content. This command allows you to run multiple validation tools serially or concurrently.

The results of validation can either be outputted to a log file or to the terminal.

Running the validate command will look something like this:

$ prm validate --codedir . --group syntax_validation
3:49PM INF Found tool group: syntax_validation
3:49PM INF Validating with the metadata-json-lint tool
3:49PM INF Validating with the puppet-syntax tool
3:49PM INF Validating with the puppet-lint tool
TOOL NAME | VALIDATION EXIT CODE | FILE LOCATION
---------------------+----------------------+--------------------------------------------------------------------------------
puppet-syntax | 1 | .prm-validate/syntax_validation/puppet-syntax_2022_April_26_16-49-59.log
metadata-json-lint | 0 | .prm-validate/syntax_validation/metadata-json-lint_2022_April_26_16-49-59.log
puppet-lint | 0 | .prm-validate/syntax_validation/puppet-lint_2022_April_26_16-49-59.log
3:49PM ERR Validation returned 1 error

Check out the docs for more information on PRM and how to use these new features!

Feedback 🗣

If you try PRM and notice any bugs, have ideas for future features, or would like to learn more about the development, feel free contact anyone from our team via community Slack or submission of a GitHub issue.

Blog: Hello From The CAT Team 👋

Fri, 22 Apr 2022 00:00:00 +0000

Hey Everyone,

Firstly let me introduce myself, I am Aaron Robinson, the Engineering Manager for the newly rebranded Content and Tooling Team (CAT). You may have known us under previous guises such as DevX and IAC, but following on from some organizational changes we have merged these two teams together and rebranded as the Content and Tooling Team.

What do we do?

The CAT team are responsible for maintaining and developing a number of supported modules and tools in the Puppet ecosystem. It is our mission to empower partners, contributors and users to create and consume high quality content.

Who are we?

The Content and Tooling team is based in the UK, with the majority of our engineers in the Belfast office and one remote engineer in mainland UK. Our Product team is based out of our Portland HQ and we regularly collaborate with them to ensure our strategic direction is aligned.

Check out below for some quick intros to the team!

Aaron Robinson - Engineering Manager

I joined Puppet in January this year, moving from an Infrastructure Management role(Network Engineering to be specific) and have over 10 years experience in the tech industry. I am a sports fanatic and I can usually be found in the stands supporting my local football(soccer if you are in the US) team or the Ulster Rugby team. If not there, you will find me on the golf course or the squash court.

You can find me on GitHub.

Craig Gumbley - Principal Software Engineer

Hello! I’m Craig, I ❤️ open source & automation. I’m excited to be starting a new journey with the Content and Tooling team. Outside of work, I enjoy listening to and making music, coding and just about anything that is technology related.

You can find me on GitHub and Twitter.

Paula Muir - Senior Software Engineer

Hey! I’m Paula and I have been with many different variations of this team over the past 6 years. I have 2 small children that keep me super busy. Any downtime I have I like to try to get creative and attempt baking, sewing, and some gardening.

You can find me on Github.

David Swan - Software Engineer

Hey, you all should hopefully know me by now, but in case you don’t I’m David Swan, a software engineer within the CAT team. I’ve been a member of the main team and its spin offs for a few years now, back when it was simply called Modules. Anyway, I’ll be continuing to work on the team for the foreseeable future, so please don’t hesitate to reach out to me whenever you need.

You can find me on GitHub.

Lukas Audzevicius - Associate Software Engineer

Hello, I’m Lukas Audzevicius, an associate level software engineer who joined Puppet in January. I graduated last year (2021) and although my journey in the IT industry has been short so far, I hope I will be able to learn and grow more working in Puppet. Outside of work, some of my most common pastimes are gaming and listening to music. I look forward to having the opportunity to work with Puppet and the community to build great things!

You can find me on GitHub.

Peter Murphy - Software Engineer Intern

Hi, my name is Peter Murphy. I am a Software Engineering student at Queen’s University Belfast and I’m working for Puppet on a year-long internship. I’ve always had an interest in computers and software, so I’m very happy to be working in a field I enjoy. Outside of work, I have many hobbies including scuba diving, badminton and hiking.

You can find me on GitHub.

Micah Wilson - CAT Product Manager

Hi, I’m Micah Wilson. I’m the PM for the CAT team, the Forge team, and the PIE team. I live in Portland, OR, more specifically way out in the suburb of Damascus. Outside of work I enjoy playing volleyball, doing OrangeTheory, and playing with my adorable French Bulldog, Toulouse.

The Future

Now with those intros out of the way we can get on with the good stuff.

With the relaunch of our team, we are currently detailing our strategy moving forward and will be continuing work on PRM which, alongside PCT, promises to provide a cleaner and lighter future for our development tools.

On the modules side we have been focusing on stability and reviewing the supported modules on the forge. Over the coming weeks will be reducing the number of supported modules to reflect recommendations proposed in the review. This will allow us to focus on the modules that we feel have true value and are aligned with our customers and support teams.

We are hoping to bring forward some new partnerships with a number of providers to look to establish partner modules and further evolve those we have so keep an eye out for some details on these as we progress through 2022.

Finally, with our rebrand and now our relaunch complete, we will be reinstating Community Day on Monday 16th May. For those of you unfamiliar with Community Day, it occurs each Monday. During this time we will triage and address new issues and pull requests from the community and not from our internal workslate. These are then prioritized and either actioned or put into the backlog for further discussion.

As the team gets up to speed you will see a lot more community engagement from the team and we will look to build on this and work with the community to build a brighter future for content at Puppet.

Thank you for taking the time to read this blog, we look forward to working alongside our community again and rebuilding that relationship.

Feel free to reach out and say hello via the community slack channel and I anticipate many great ideas coming forward to us on how we enhance and build the experience with our modules and tools.

Aaron.

Blog: Puppet Runtime Manager 0.1.0 Release 🚀

Mon, 31 Jan 2022 00:00:00 +0000

We are proud to announce the first release for the new tool for executing arbitrary commands against your code in a Puppet runtime: The Puppet Runtime Manager (PRM).

The Puppet Runtime Manager abstracts away the Puppet Ruby development environment to enable you to execute against and validate content for any Puppet Product that you might author!

Installing PRM 💾

To get started, you can either download the release from the PRM repo or use our installer scripts:

Unix Systems
Windows Systems

curl -L https://pup.pt/prm/install.sh | sh

iex "&{ $(irm https://pup.pt/prm/install.ps1); Install-Prm }"

This will install the latest release of PRM to ~/.puppetlabs/prm.

PRM is Experimental!

PRM is currently in an EXPERIMENTAL phase and feedback is encouraged via our Discussions page on GitHub.

Telemetry & PRM

By default, PRM collects some telemetry during each run. This information is non-identifying and does not include any personal or organizational details.

For more information, see our documentation on Telemetry.

If you’d like to install PRM without telemetry, you can use these scripts:

Notel on Unix Systems
Notel on Windows Systems

curl -L https://pup.pt/prm/install.sh | sh -s -- --no-telemetry

iex "&{ $(irm https://pup.pt/prm/install.ps1); Install-Prm -NoTelemetry}"

Prerequisites & Support

In this initial release, PRM only supports one backend for running tools: Docker. You will need to be able to use Docker locally to pull, build, and run container images for this release (though PRM handles all those steps for you).

Right now, only Linux containers in Docker are supported, but the Docker backend is functional on Windows, Linux, and MacOS systems.

A future release will add a local execution backend, enabling you to just use your locally installed Puppet Agent for the runtime.

If there are other backends you’d like to see supported, please raise a discussion with us!

Docs in the Terminal 📃

PRM has a pair of robust help systems built in for reviewing reference, concept, and narrative documentation.

To see the help for any command or subcommand:

Use the help subcommand
Use the help flag

prm help # Displays the help for PRM itself
prm help exec # Displays the help for the prm exec command

prm --help # Displays the help for PRM itself
prm exec --help # Displays the help for the prm exec command

To read the concept and narrative documentation locally instead of online:

prm explain --list
prm explain # Identical to prm explain --list

prm explain --category concept # Only list concept docs
prm explain --tag tools # Only list docs about tools
prm explain --category concept --tag tools # Multi-filter

prm --help # Displays the help for PRM itself
prm exec --help # Displays the help for the prm exec command

For more detailed information on the explain subsystem and documentation, see our narrative document on it.

Run Tools Against Your Content 🏃🏻‍♀️

PRM can run tools against any type of a Puppet Product project. You can run r10k against Puppet control repos, validate EPP files in Puppet modules, rubocop Bolt projects, and more!

We’ve shipped with a subset of tools you can use:

EPP
Heira EYAML
metadata-json-lint
onceover
puppet-lint
puppet-strings
puppet-syntax
r10k
ra10ke
rubocop
rspec-puppet

These tools are currently maintained by the DevX team in the fantastic-adventure repository. If you’d like to learn more about how they’re constructed or how to develop your own, check out our Anatomy of a Tool guide.

Since PRM tools are external to the PRM code base, Puppet Product teams, Partners and the wider OSS Community can publish their own tool packages without input or effort from the DevX team. This enables a single tool to support many products.

Example: Generate Reference Docs for puppetlabs-acl

This example is a shortened version of our Quick Start Guide, which includes a more detailed walkthrough and includes configuring PRM beyond the defaults.

Before we generate the reference documents for a Puppet module, we might want to ensure that we have full coverage for our module.

Assuming we’ve already cloned a copy of [https://github.com/puppetlabs/puppetlabs-acl] locally to ~/code/modules/puppetlabs-acl, we can run the following command:

prm exec puppetlabs/puppet-strings --codedir ~/code/modules/puppetlabs-acl

4:54PM INF Creating new image. Please wait...
4:54PM INF Code path: ~/code/modules/puppetlabs-acl
4:54PM INF Cache path: ~/.pdk/prm/cache
4:54PM INF Additional Args: []
Files: 5
Modules: 2 ( 0 undocumented)
Classes: 4 ( 0 undocumented)
Constants: 25 ( 0 undocumented)
Attributes: 9 ( 0 undocumented)
Methods: 49 ( 0 undocumented)
Puppet Data Types: 0 ( 0 undocumented)
Puppet Data Type Aliases: 0 ( 0 undocumented)
Puppet Classes: 0 ( 0 undocumented)
Puppet Types: 1 ( 0 undocumented)
Puppet Providers: 1 ( 0 undocumented)
Puppet Functions: 0 ( 0 undocumented)
Puppet Defined Types: 0 ( 0 undocumented)
Puppet Plans: 0 ( 0 undocumented)
Puppet Tasks: 0 ( 0 undocumented)
100.00% documented
4:54PM INF Tool puppetlabs/puppet-strings executed successfully

Behind the scenes, PRM built a docker container on the fly from the tool definition for puppet-strings and then executed it against the specified directory. The output we see without the PRM log prefix is what the docker container’s stdout returned; in this case, the documentation results for the ACL module.

We can also pass arbitrary arguments to the tool; to generate the reference documentation, we need to specify that we want to generate the document and set the format to markdown. This will create (or update if it exists) the REFERENCE.md file in the module’s folder.

prm exec puppetlabs/puppet-strings --codedir ~/code/modules/puppetlabs-acl --toolArgs "strings generate --format markdown"

4:54PM INF Creating new image. Please wait...
4:54PM INF Code path: ~/code/modules/puppetlabs-acl
4:54PM INF Cache path: ~/.pdk/prm/cache
4:54PM INF Additional Args: [strings generate --format markdown]
Files: 5
Modules: 2 ( 0 undocumented)
Classes: 4 ( 0 undocumented)
Constants: 25 ( 0 undocumented)
Attributes: 9 ( 0 undocumented)
Methods: 49 ( 0 undocumented)
Puppet Data Types: 0 ( 0 undocumented)
Puppet Data Type Aliases: 0 ( 0 undocumented)
Puppet Classes: 0 ( 0 undocumented)
Puppet Types: 1 ( 0 undocumented)
Puppet Providers: 1 ( 0 undocumented)
Puppet Functions: 0 ( 0 undocumented)
Puppet Defined Types: 0 ( 0 undocumented)
Puppet Plans: 0 ( 0 undocumented)
Puppet Tasks: 0 ( 0 undocumented)
100.00% documented
4:54PM INF Tool puppetlabs/puppet-strings executed successfully

While the output of Puppet Strings itself isn’t any different, we can check the timestamp on the REFERENCE.md file and verify that it was just updated (or created if it didn’t already exist).

So! That’s how you run a PRM tool against a folder and can send the tool specific arguments, too!

Coming Soon❗

As mentioned earlier, this is the initial release for PRM; we’re not done!

Our next planned release, 0.2.0, will add the validate functionality you may remember from the PDK. In short, this will enable you to run multiple tools against your code in parallel and report back to you both pass/fail and detailed failure notes (if any). We are targeting this release for late March/early April.

After that, our 0.3.0 release will add local execution as a backend, removing the dependency on Docker so long as you have the Puppet Agent installed locally.

In the meantime, we will be improving and adding tools; if you add any yourself, we’d love to hear about it and highlight them in future blog posts!

Til next time!

~ The DevX Team

Blog: A Fond Farewell For Now to Community Day & Norman

Wed, 20 Oct 2021 00:00:00 +0000

IAC Community Day

Background

The Infrastructure Automation Content (IAC) team formed from the merger of 4 core content-focused teams (Modules, Windows, Cloud & Containers, and Networking), maintaining forty-five open-source supported Puppet modules and over a dozen tools that help reduce the content maintenance and support costs.

A crucial part of Puppet’s success has always been the support of the community. The IAC team, like with all module teams before, would have had a dedicated engineer or engineers assigned to community triage, usually on a rotational basis.

The triage rota consumed 20% of the team’s engineering resources. It was challenging due to:

Lack of domain knowledge
Work spilling over either into or out of triage during rotation
Impact to other team members not on rotation
Engineers left feeling drained, low self-esteem
Engineers dreading rotating onto triage.

Community Day (Introduced in November 2020)

Community day came about as part of reviewing of our triaging commitments and chatting with the team listening to their concerns.

As mentioned, 20% of the team’s engineering resourcing and effort was already going into the triage rotation. So we asked ourselves the question:

What if we focused this 20% effort and had the whole team take part in a full day community triage?

Benefits:

Upskilling
Teamwork
Reduced distractions
Greater community focus
Starting the week off with a win
Productivity within the team increased (hard to believe even for us, at first)

Not only have we been able to address the backlog of community requests across forty-five supported open source modules (plus over a dozen tooling repos), the team have been able to focus on feature delivery.

Since switching to the new community day, over the last eleven months, the team have:

Handled† 1536 issues
Performed 205 module releases
Released 30 tooling updates

† Handled meaning a team member has either:

Closed a PR or Jira

Merged community a PR

or provided feedback on a community PR

By committing to community day and limiting distractions to 1 day, the team have also delivered:

A new Cloud CI
1. With multi-node testing support
2. New automated workflow
3. Internal developer environment support
Cisco Network device enhancement
New DSC builder generating >300 DSC PowerShell modules uploaded on the forge
3 OS certifications
Certified all the supported modules again Puppet 7, including:
1. Raising and testing fixes for numerous issues in Puppet 7 nightly agent builds prior to release
Rolled out the Trusted Contributor program
Improved our community communication with a weekly blog post, enhancing our community reporting

All in all, the IAC community day has not only enabled the team to reconnect with the community, address the content backlog, and upskill across the team; it has also improved team productivity by reducing distraction and increasing team morale. Mondays are, at the best of times, a tough day. Starting a week off with a win? Priceless.

Sadly, Community Day is coming to an end. Due to a significant reduction in resourcing, it is no longer possible to provide the technical support needed to run a dedicated community support day.

The content maintained by the IAC team is still of high importance to Puppet. Puppet would like to encourage our community to reach out via our public slack channels for assistance. Our PM team will help prioritize issues raised.

Farewell, Norman 💜

This week we have to say farewell to Norman, the Engineering Manager of the IAC and DevX Teams. This is a particularly sad goodbye as many of us, past and present, consider Norman one of the best leaders we have had the privilege of working with.

In just under three years, Norman led us to some fantastic achievements and kept us motivated and moving forward through turbulent times. Norman has shown unparalleled dedication to his team’s wellbeing, motivation, and personal development whilst still acting as a clarifying arbiter between the high-level vision and plans of the organisation. It was a tough balancing act that he seemed to be able to do, with his usual 110% dedication he never falters from.

Norman motivated and enabled us to become far better engineers and people than we were prior to being taken under his wing. The remit of the IAC team is a tough one, and one that can often be underappreciated, but Norman always fought our corner and sung our praises to anyone who stood still long enough to listen. Norman was great at suggesting and soliciting us to suggest new processes / ideas (Community Day being a prime example), that would make us more efficient and motivated, as a team. We benefited from his wealth of knowledge, experience (both technical and managerial) and genuine care for his colleagues and team.

We’d like to thank Norman for his hard work and support over these nearly three years. The IAC and DevX teams owe him a huge gratitude and we’ll miss him, as we bid farewell and good luck in your next venture!

Blog: The future of the Puppet Developer Kit (PDK)

Tue, 14 Sep 2021 00:00:00 +0000

The Puppet Developer Kit was launched in 2017 to help our users develop high quality modules faster and over the years, it has seen regular development towards that goal.

It was designed to be a single install that would provide everything you needed to write, lint, test and publish a Puppet module — and for the most part, that’s what it did.

What worked

The PDK did exactly what it set out to do. It enabled users of Puppet and its ecosystem — regardless of experience level — to create Modules that were testable, deployable, and shareable. It provided functionality to keep those modules running against all supported Puppet versions, as well as the core Gemset to run all the features built into the template.

What didn’t work

In order for all this to work, the PDK hinged on that single template which was maintained by the Puppet Developer Experience (PDE) team.

Since its launch, the PDK has come to support the creation of more than just modules and classes. With each successive addition, the template and the packaging system had to support more dependencies, and these dependencies needed to be monitored and updated in line with Puppet agent support and things like security fixes. This overhead meant the PDK couldn’t move forward quickly in supporting the wealth of content our users wanted. And every new feature made the template system that much more complex and awkward to customize for your own needs.

The path forward

The DevX strategy is to build a new PDK while maintaining the existing PDK at version 2.x. What this means is that updates to the PDK 2.x will be limited to bug fixes and updates to support the latest Puppet gem.

We will be designing and building multiple tools to cover the functionality of the existing PDK:

Content creation
Linting / Validation
Testing
Packaging and sharing
Runtime management
Each tool will be released ASAP for early feedback and iteration. They will be designed to work independently of each other, giving you the choice to download and install the parts of the Kit that you need.

Puppet Content Templates

You may already be aware of the DevX team’s first part of this strategy: Puppet Content Templates (PCT).

PCT expands on the original PDK’s “new” functionality enabling Puppet, Puppet content creators, and our customers to create their own templates and share them with their teams. No longer do we have a single one-size-fits-all template. Instead we have many small and composable templates that allow you to mix & match to make your modules everything you want — and only what you want.

If you’re new to Puppet, PCT ships with the common templates:

Module
Class
Type

This functionality was available in PDK, but with our new approach, we’ve been able to quickly build templates for other Puppet content:

Control Repo
Resource API Provider
Bolt Projects
Bolt Plans
Bolt Tasks

Note

At the time of writing, we are in the process of shipping PCT 0.3.0 which adds the ability to package and install new templates. This makes it possible to share the templates with others on your team by just distributing a tarball.

In the near future we also plan to allow for the publication of templates to the Puppet Forge for sharing, download and installation. If you want all your modules to include GitHub actions for auto-publishing to the Forge when you tag a release, you’d just install that template. If you prefer GitLab or CircleCI instead, you’d choose the appropriate template. And if one doesn’t exist that does what you need, then you could write it yourself and contribute it to the community.

Note

PCT is in active development and is pre-release; therefore it is not recommended for production use at this time.

To find out more on PCT, you can watch James Pogran’s coverage of it on Pulling the Strings LIVE.

Technical Choices

To build these new tools, we have consciously chosen to move away from Ruby as a platform and are instead developing these CLIs in Go. This allows us to ship a compiled binary which is faster for you to run, and faster for us to build and test. This results in our team’s ability to iterate faster and ship new rich features more often. It also means that your experience of creating a new module happens before your finger has lifted from the enter key.

Ruby will always be present when testing and validating. Puppet and its ecosystem are built upon it. Our new tooling will include a system that will provide the PDK functionality of a Puppet Runtime sandbox, with you choosing which version of Puppet to install and run our tooling against.

The aim for the new tools is to make them loosely coupled to Puppet, while still providing a clear workflow that allows you to build great Puppet content. This means that you will not be forced to update when the pdk-templates are updated, or to include functionality in your code that you don’t want or need. We hope this approach will provide flexibility for different development choices and will help new users understand wider Puppet concepts without the sometimes intrusive guardrails provided by the current PDK.

Finally, we want the tools to be extensible. With PCT, you can install new templates. With our validation tool, we want content creators to be able to plug in new validators, and to write validation tools for testing their content. And with our Packaging and Sharing tool, we want you to be able to share content to multiple locations, including the Forge.

With this flexible plugin architecture, this could mean eventually supporting Onceover on your control repository before upgrading to a new version of Puppet, or being able to fire up Corey’s Puppet Debugger on a previous version of Puppet to figure out the intended behavior of that old module you’ve inherited.

Feedback

Feedback is so important to us at this early stage of design and development. The DevX team would love to engage with our community about our roadmap and we hope that you can see yourself using these tools in the future. Your feedback can have a real impact on what we build.

To get in touch, you can find us on the #office-hours channel within Puppet Community Slack every Monday, alternating weekly between about 1pm and 5pm GMT.

Out of those hours, please get involved in our discussions at Discussions · puppetlabs/pct · GitHub.

Blog: Puppet Content Templates 0.1.0 🚀

Fri, 04 Jun 2021 00:00:00 +0000

PDK Puppet Content Templates 0.1.0 released ✨

We are proud to announce the first release for the new PDK templating system: Puppet Content Templates (PCT).

Puppet Content Templates codify a structure to produce content for any Puppet Product that can be authored without direct oversight of the Puppet PDK team.

To get started download the release from the pct repo. A step by step guide is included in the README.

PCT is currently in an EXPERIMENTAL phase and feedback is encouraged via pct/discussions. ⚠️

Create all types of content 🎨

A PCT can create any type of a Puppet Product project. You can create Puppet control repos, Puppet modules, Bolt projects, and more!

A PCT can also create one or more independent files to be used inside a project. You can create Puppet classes, facts, functions, or other tooling configuration files such as CI files, gitignores, or rubocop.yml.

Since a PCT is external to the PDK code base, Puppet Product teams, Partners and the wider OSS Community can publish their own content without input or effort from the DevX team. This enables a single tool to support many products.

Building blocks 🧱

A set of individual PCTs can be combined to produce something as simple as a Puppet module with several Puppet classes or as complex as a complete Puppet Control repo with roles and profiles ready to deploy.

We want to enable a building block experience. An experience where you can pick and choose from an array of templates to build the content you want. We aim to be like your favorite building block toy, where you choose your components and end up with a wonderful structure.

An opt-in experience, where you choose what is added to your content, instead of dealing with a long list of files you’ll never use.

Customizable 📐

Each PCT comes with parts that can be controlled via configuration. These parts have common defaults, but can be changed to suit your environment. The idea is to enable quick deployment of templates, but allow customization to suit your needs.

This user can customize how PCTs operate by using YAML files and never need to know how to make a template.

If you want a more drastic change, you can copy the PCT and add your own files and changes, then use that PCT without having to get a pull request approved.

Blog: Run multi-node module tests with cloud continuous integration (CI) in Google Cloud Platform (GCP)

Fri, 07 May 2021 00:00:00 +0000

Multi-node modules are modules which use multiple nodes to run integration tests. Integration testing is where you set up multiple Virtual Machines (VM) or containers, and test interactions between them. For example, this could be:

Installing Puppet Enterprise (PE) and multiple Puppet agents.
Setting up an NTP server and registering NTP clients.
Installing open source Puppet Server and multiple Puppet agents.

The order you perform integration tests is important, and you need to be able to run a test on an individual system.

Running multi-node modules in GCP involves the following steps:

Provision a node using a Bolt task.
Set up a multi-node environment for testing using a Bolt task or plan.
Run a test - you can use existing serverspec or Litmus helpers to set up any dependencies required by the module.
Teardown the machine using a provision task.

This guide walks you through each step, and then provides examples of how your code would look in different environments.

Provision a node

You can use Litmus and a Bolt task to provision a VM or a container. The provision module’s available tasks spin up the test environment. When run, it creates a litmus_inventory.yaml file that allows Bolt and serverspec to communicate with that VM.

Running tasks in the provision module allows you to add arbitrary key/pair values to the Bolt inventory file. The Bolt variables allows you to have multiple labels associated with a single machine. For example:

Example_websphere_provision_plan

Set up a multi-node environment for testing

To step up Litmus for your acceptance tests, you need to:

Install a Puppet agent — you can do this using a Bolt task.
Install a module — use a rake task that uses Puppet Development Kit (PDK) to build the module for testing, and install it on the target VM.
Install spec/spec_helper_acceptance_local.rb — this may require extra setup to test a module. You can use puppetlabs/puppetlabs-apache.

To set up integration testing, use a Bolt plan and inventory variable to add labels. You can then run Puppet code against a specific system. You can use puppetlabs-websphere_application_server

An example provision plan looks like:

plan websphere_application_server::provision_machines(
Optional[String] $using = 'abs',
Optional[String] $image = 'centos-7-x86_64'
) {
# provision machines, set roles
['server', 'appserver', 'dmgr', 'ihs'].each |$role| {
run_task("provision::${using}", 'localhost', action => 'provision', platform => $image, vars => "role: ${role}")
}
}

You can use spec/spec_helper_acceptance_local.rb if you need to test a module — puppetlabs-websphere_application_server

Run a test

To identify which tests to run in an integeration environment, you can use rspec labelling and rake tasks.

You can label tests using rspec labeling — tag the test as an integration and keep everything else can stay the same. For example:

describe 'Install the websphere dmgr', :integration do
before(:all) do
@agent = WebSphereHelper.dmgr_host
WebSphereInstance.install(@agent)
WebSphereDmgr.install(@agent)
end
it 'is installed' do
expect(WebSphereHelper.remote_file_exists(@agent, WebSphereConstants.dmgr_status))
expect(WebSphereHelper.remote_file_exists(@agent, WebSphereConstants.ws_admin))
end
end

To run a rake task, add to the Rakefile of the module. For example:

require 'rspec/core/rake_task'
namespace :websphere_application_server do
RSpec::Core::RakeTask.new(:integration) do |t|
t.pattern = 'spec/acceptance/**{,/*/**}/*_spec.rb'
t.rspec_opts = "--tag integration"
end
end

To target a test at a specific VM or container, you can use helper methods. You need to do this if you are using serverspec or Litmus.

The examples below show how to filter to get either a single VM or container, or how to retrieve multiple targets. puppetlabs-websphere_application_server

puppetlabs-kubernetes

To identify a target node, the code would look like:

 context 'application deployment' do
before(:all) { change_target_host('controller') }
after(:all) { reset_target_host }
it 'can deploy an application into a namespace and expose it' do
run_shell('KUBECONFIG=/etc/kubernetes/admin.conf kubectl create -f /tmp/nginx.yml') do |r|
expect(r.stdout).to match(/my-nginx created\nservice\/my-nginx created\n/)
end
end

To only run on tests labelled as ‘integration’, use the following command:

bundle exec rake websphere_application_server:integration

Teardown the machine

To teardown all the provisioned machines, use the following provision task:

bundle exec rake litmus:tear_down

Multi-node testing examples

Multiple Puppet agents

A setup where you have multiple Puppet agents, with a module installed and tests run.

Commands

bundle install --path .bundle/gems/ --jobs 4
bundle exec rake spec_prep
bundle exec bolt --modulepath spec/fixtures/modules plan run ntp::provision_gcp
bundle exec rake litmus:install_agent
bundle exec rake litmus:install_module
bundle exec rake ntp:integration
bundle exec rake 'litmus:tear_down'

Example puppetlabs-ntp

Puppet server and multiple agents

A setup where you have Puppet Server and multiple Puppet agents.

You can use Bolt plans for provisioning multiple nodes, with roles tagged to each of it.

Puppet has a new task that installs an open source Puppet server in the provision module.

Commands

bundle install --path .bundle/gems/ --jobs 4
bundle exec rake spec_prep
bundle exec bolt --modulepath spec/fixtures/modules plan run kubernetes::provision_cluster
bundle exec bolt --modulepath spec/fixtures/modules -i ./spec/fixtures/litmus_inventory.yaml plan run kubernetes::puppetserver_setup
bundle exec rake litmus:install_agent
bundle exec rake litmus:install_module
bundle exec rake kubernetes:integration
bundle exec rake 'litmus:tear_down'

Example puppetlabs-kubernetes

PE and agents

A setup where you have a PE server and multiple Puppet agents.

You can use Bolt plans for provisioning multiple nodes with roles tagged to each of it.

There is a task to install PE in the puppet-deploy_pe module. provision_master/agents

Commands

bundle install --path .bundle/gems/ --jobs 4
bundle exec rake spec_prep
bundle exec bolt --modulepath spec/fixtures/modules plan run ntp::provision_gcp
bundle exec bolt --modulepath spec/fixtures/modules -i ./spec/fixtures/litmus_inventory.yaml plan run ntp::pe_server
bundle exec bolt --modulepath spec/fixtures/modules -i ./spec/fixtures/litmus_inventory.yaml plan run ntp::pe_agent
bundle exec rake litmus:install_module
bundle exec rake ntp:integration
bundle exec rake 'litmus:tear_down'

Example puppetlabs-ntp

An example plan that installs a PE server:

plan ntp::pe_server(
Optional[String] $version = '2019.8.5',
Optional[Hash] $pe_settings = {password => 'puppetlabs'}
) {
# identify pe server node
$puppet_server = get_targets('*').filter |$n| { $n.vars['role'] == 'ntpserver' }
# install pe server
run_plan(
'deploy_pe::provision_master',
$puppet_server,
'version' => $version,
'pe_settings' => $pe_settings
)
}

An example plan that installs a Puppet agent:

plan ntp::pe_agent() {
# identify pe server and agent nodes
$puppet_server = get_targets('*').filter |$n| { $n.vars['role'] == 'ntpserver' }
$puppet_agent = get_targets('*').filter |$n| { $n.vars['role'] == 'ntpclient' }
# install agent
run_plan(
'deploy_pe::provision_agent',
$puppet_agent,
'master' => $puppet_server,
)
}

For more GitHub Action workflow examples, see the following:

Thanks TP for the valuable work on integration testing. Thanks Marty for the work on installing PE on cloud CI.

Blog: Converting to use Puppetized DSC Modules

Tue, 26 Jan 2021 00:00:00 +0000

What’s new with DSC and Puppet?

To check out our reasons on why we think converting your module is a good idea, check out these blog posts written by Michael:

The blog posts cover information including:

Why we are doing this
Details about the builder
How to Pupperize your Powershell module
How to use your Puppetized module

Seriously though what is there to not love?

Minimal module maintainence costs
Automated module building
Property by property reporting
VSCode Extension with intellisense
We have 68 modules on the Forge in the DSC namespace
Converting is relatively straight forward

So how can I do this?

Old method:

mod 'puppetlabs-dsc', '1.9.4'

New method:

Add entries for puppetlabs-pwshlib and the dsc-* modules whose resources you want to use to your puppetfile. The entries on the list below are the closest match for the resources found in the old puppetlabs-dsc module; while we specified versions here, remember: there have been releases to the DSC modules since the puppetlabs-dsc last had its resources updated, so there are newer versions you may want to take advantage of (just be sure to test thoroughly when upgrading)!

Note: You want to very specifically only add the necessary dsc-* module pins to your puppetfile; if you’re not using any of the resources contained in a module, there’s no need to specify it and take on the overhead of shipping the code around.

mod 'puppetlabs-pwshlib', '0.7.1'
mod 'dsc-activedirectorycsdsc', '3.1.0-0-0'
mod 'dsc-auditpolicydsc', '1.4.0-0-1'
mod 'dsc-certificatedsc', '4.4.0-0-0'
mod 'dsc-computermanagementdsc', '6.2.0-0-0'
mod 'dsc-dfsdsc', '4.3.0-0-0'
mod 'dsc-networkingdsc', '7.0.0-0-0'
mod 'dsc-officeonlineserverdsc', '1.2.0-0-0'
mod 'dsc-securitypolicydsc', '2.7.0-0-0'
mod 'dsc-sharepointdsc', '3.2.0-0-0'
mod 'dsc-sqlserverdsc', '12.3.0-0-0'
mod 'dsc-storagedsc', '4.5.0-0-0'
mod 'dsc-systemlocaledsc', '1.2.0-0-0'
mod 'dsc-xactivedirectory', '2.24.0-0-0'
mod 'dsc-xazure', '0.2.0-0-0'
mod 'dsc-xazurepack', '1.4.0-0-0'
mod 'dsc-xbitlocker', '1.4.0-0-0'
mod 'dsc-xcredssp', '1.3.0-0-0'
mod 'dsc-xdatabase', '1.9.0-0-0'
mod 'dsc-xdefender', '0.2.0-0-0'
mod 'dsc-xdhcpserver', '2.0.0-0-0'
mod 'dsc-xdisk', '1.0.0-0-0'
mod 'dsc-xdismfeature', '1.3.0-0-0'
mod 'dsc-xdnsserver', '1.11.0-0-0'
mod 'dsc-xexchange', '1.27.0-0-0'
mod 'dsc-xfailovercluster', '1.12.0-0-0'
mod 'ddsc-xhyper_v', '3.16.0-0-0'
mod 'dsc-xinternetexplorerhomepage', '1.0.0-0-0'
mod 'dsc-xjea', '0.2.16-6-0'
mod 'dsc-xmysql', '2.1.0-0-0'
mod 'dsc-xpendingreboot', '0.4.0-0-0'
mod 'dsc-xphp', '1.2.0-0-0'
mod 'dsc-xpowershellexecutionpolicy', '3.1.0-0-0'
mod 'dsc-xpsdesiredstateconfiguration', '8.5.0-0-0'
mod 'dsc-xremotedesktopadmin', '1.1.0-0-0'
mod 'dsc-xremotedesktopsessionhost', '1.8.0-0-0'
mod 'dsc-xrobocopy', '2.0.0-0-0'
mod 'dsc-xscdpm', '1.2.0-0-0'
mod 'dsc-xscom', '1.3.3-0-0'
mod 'dsc-xscsma', '2.0.0-0-0'
mod 'dsc-xscspf', '1.3.1-0-0'
mod 'dsc-xscsr', '1.3.0-0-0'
mod 'dsc-xscvmm', '1.2.4-0-0'
mod 'dsc-xsmbshare', '2.1.0-0-0'
mod 'dsc-xsqlps', '1.4.0-0-0'
mod 'dsc-xtimezone', '1.8.0-0-0'
mod 'dsc-xwebadministration', '2.5.0-0-0'
mod 'dsc-xwebdeploy', '1.2.0-0-0'
mod 'dsc-xwindowseventforwarding', '1.0.0-0-0'
mod 'dsc-xwindowsrestore', '1.0.0-0-0'
mod 'dsc-xwindowsupdate', '2.7.0-0-0'
mod 'dsc-xwineventlog', '1.2.0-0-0'
mod 'dsc-xwordpress', '1.1.0-0-0'

Note that the versions for the dsc-* modules are all 5-digits long. The first four digits are a match for the puppetized PowerShell module’s own version - so 1.4.0-0-0 represents a PowerShell module at version 1.4.0.0 on the PowerShell Gallery. The fifth digit is the Puppetized Build Increment for that version; that goes up whenever we rebuild the module to fix a bug or add a feature to the Puppet interface to the DSC Resource. In general, you’ll almost always want the latest build of a module. So even though we specify Puppet Build Increments of 0 in the list above, make sure to choose the latest build for the PowerShell module version you want to use.

A few caveats

The following modules cannot be automatically puppetized at the same version because they referenced commit SHAs instead of versions discoverable on the gallery; instead, we recommend using the next released stable version after those SHA’s, as listed below:

Note: These are close but not precise replacements:

mod 'dsc-fsrmdsc', '2.5.0-0-0'
mod 'dsc-iscsidsc', '2.0.0-0-0'
mod 'dsc-wsmandsc', '2.4.0-0-0'

Here are links to the specific commit SHAs that have been referenced:

Note: puppetized as dsc/wminamespacesecurity due to the name of the module changing when pushed to the PowerShell Gallery vs the old code repository.

mod 'dsc-wminamespacesecuritydsc', '0.3.0-0-0'

Again here is a link to the specific commit SHA that has been referenced:

WmiNamespaceSecurityDsc

Guidance on troubleshooting

Coincidentally, Michael has just published a blog post on how to troubleshoot for DSC.

Check it out on our blog page!

Some things come for free!

We suggest using the Puppet VSCode extension as it comes built in with an auto-completer. This will help you keep your pins up to date with the puppetfile resolver.

What else is coming

As part of our next phase we plan to flesh out and update our testing infrastructure for this module on our road to the 1.0.0 release of Puppet.Dsc. In the pipeline, we still plan on getting nightly builds into place to ensure that there is never more than a 24 hour lag between published Powershell modules with DSC Resources and our Puppetized modules on the Forge.

As always if you have any questions don’t hesistate to reach out to us via Slack or emailing and we will be as accomodating as we can be!

Blog: Troubleshooting Puppetized DSC Modules

Mon, 25 Jan 2021 00:00:00 +0000

[An edited version of this post was also published on the OSPAssist Portal as Troubleshooting Puppetized DSC modules and Advanced troubleshooting: dig deeper with pry]

Authoring Puppet manifests using the new Puppetized DSC Resources is, like using any other Puppet modules, a few steps and a deploy away for most use cases. Sometimes you will run into a problem trying to apply a manifest containing one or more puppetized DSC resources. What could be causing the problem?

Is it a problem with the builder? The Puppet representation of the DSC Resource could not match the API surface, resulting in misgenerated types, enums or method calls.

Could the problem be the DSC Resource itself? The PowerShell DSC code could not work, which would be hidden underneath the Puppet Ruby exception stacktrace.

What happens if it’s Puppet? There could be an error in the Ruby code that translates the DSC resource back and forth between Puppet and PowerShell, in which case the issue lies in the pwshlib module.

Somewhere in between? How do you tell which symptom is which problem? Where do you begin to troubleshoot this? This blog post will help you find your way foreward.

Basic Troubleshooting

The first thing to note is that there’s actually a basic troubleshooting guide for Puppet.Dsc. That guide covers finding problems like type errors or incorrect enums in the module builder, invocation errors during a Puppet run, and problems with the tanslation layer in pwshlib. There is a lot of good information there, so we won’t cover those areas again here.

The guide also covers running Puppet in debug mode, which you will need to know about for the next couple sections. In short, if you append the --debug flag to the end of your Puppet invocation, you’ll get a ton of additional information out of the run. Please make sure to read the guide to ensure you have a reference point before we proceed here.

Advanced Troubleshooting

So, you have a Puppet manifest with one or more Puppetized DSC Resource declarations in it. Things are not working and the basic troubleshooting guide didn’t solve the problem.

The first place to start is to validate that what you declared in the manifest was translated correctly to the PowerShell DSC code to execute.

Validating manifests and the PowerShell hash

When you specify a DSC Resource in your Puppet manifest, behind the scenes Puppet converts that into PowerShell code to pass to Invoke-DscResource. In debug mode, you can actually see what Puppet thinks you passed and that generated code.

Lets walk through an example manifest:

dsc_psrepository { 'PowerShell Gallery':
dsc_name => 'psGallery',
dsc_installationpolicy => 'Trusted',
}

If we execute Puppet in debug mode, we’ll get the following (abbreviated) output:

Debug: dsc_psrepository[{:name=>"PowerShell Gallery", :dsc_name=>"psGallery"}]: Updating: Invoking Set Method for '{:name=>"PowerShell Gallery", :dsc_name=>"psGallery"}' with {:name=>"PowerShell Gallery", :dsc_name=>"psGallery", :dsc_installationpolicy=>"Trusted"}
... (snipped for brevity)
dsc_psrepository[{:name=>"PowerShell Gallery", :dsc_name=>"psGallery"}]: Updating: Script:
... (snipped for brevity)
$InvokeParams = @{Name = 'PSRepository'; Method = 'set'; Property = @{name = 'psGallery'; installationpolicy = 'Trusted'}; ModuleName = @{ModuleName = 'C:/pathToPuppetModules/powershellget/lib/puppet_x/powershellget/dsc_resources/PowerShellGet/PowerShellGet.psd1'; RequiredVersion = '2.2.4.1'}} Try {
$Result = Invoke-DscResource @InvokeParams
} catch {
$Response.errormessage = $_.Exception.Message
return ($Response | ConvertTo-Json -Compress)
}

Breaking this apart, there are several sections of information to pull out.

The first line is Puppet telling us why it’s doing what it’s doing and with what information it has to execute. We see its the PowerShell Gallery module, using the Gallery DSC Resource and setting the installationpolicy to Trusted.

Debug: dsc_psrepository[{:name=>"PowerShell Gallery", :dsc_name=>"psGallery"}]: Updating: Invoking Set Method for '{:name=>"PowerShell Gallery", :dsc_name=>"psGallery"}' with {:name=>"PowerShell Gallery", :dsc_name=>"psGallery", :dsc_installationpolicy=>"Trusted"}

This can be more easily read as as Ruby hash:

{
:name => "PowerShell Gallery",
:dsc_name => "psGallery",
:dsc_installationpolicy => "Trusted"
}

The next part in the log to look at is the invocation parameters. This is the PowerShell code generated from the section we just examined:

$InvokeParams = @{Name = 'PSRepository'; Method = 'set'; Property = @{name = 'psGallery'; installationpolicy = 'Trusted'}; ModuleName = @{ModuleName = 'C:/pathToPuppetModules/powershellget/lib/puppet_x/powershellget/dsc_resources/PowerShellGet/PowerShellGet.psd1'; RequiredVersion = '2.2.4.1'}}

Translating that to a PowerShell hash, we can see a similar represation of the Ruby hash we just looked at. This is the exact code that will be passed to Invoke-DscResource (expanded from one line for clarity; note your ModuleName path will be system-specific):

$InvokeParams = @{
Name = 'PSRepository';
Method = 'get';
Property = @{ name = 'psGallery' };
ModuleName = @{
ModuleName = 'C:/pathToPuppetModules/powershellget/lib/puppet_x/powershellget/dsc_resources/PowerShellGet/PowerShellGet.psd1';
RequiredVersion = '2.2.4.1'
}
}

In summary, the debug logs from a Puppet execution run provide us detailed information about how your Puppet manifest was translated into PowerShell code to invoke the DSC Resource.

This leads us neatly into trying to use this information to invoke the DSC Resource outside of Puppet to debug it.

Invoking without Puppet

After inspecting the debug Puppet logs, we now have the PowerShell code Puppet was executing to try out. We can take that code and move to a test machine and walk through the execution ourselves interactively.

In a PowerShell console with administrator permissions, you’ll copy and paste the $InvokeParams line from the debug log we pulled out above. Then you’ll invoke it using Invoke-DscRsource. For example:

PS C:\> $InvokeParams = @{
>> Name = 'PSRepository';
>> Method = 'get';
>> Property = @{ name = 'psGallery' };
>> ModuleName = @{
>> ModuleName = 'C:/pathToPuppetModules/powershellget/lib/puppet_x/powershellget/dsc_resources/PowerShellGet/PowerShellGet.psd1';
>> RequiredVersion = '2.2.4.1'
>> }
>> }
PS C:\> Invoke-DscResource @InvokeParams -Verbose

Now you can evaluate the success/failure of the invocation without involving Puppet at all; this way you can determine if the values being passed were incorrect or if the DSC Resource itself is misbehaving.

If the code fails after pasting in the $InvokeParams block, then there is something wrong with the values or the hash statement. If something has gone wrong converting the values in your Puppet manifest to the InvokeParams hash being passed to Invoke-DscResource, please do file an issue with us and we’ll be sure to take a look.

If the code fails after invoking Invoke-DscResource, then there is likely a problem inside the DSC Resource itself. If it’s an issue with the DSC Resource, hopefully you’ll have enough information from the execution to put together a bug report for the upstream PowerShell module and work with that DSC resource’s authors for a resolution.

Invoking DSC Resources with PSCredentials

When your manifest specifies a PSCredential, you’ll see one or more lines above the InvokeParams declaration, like this (expanded for readability):

$febabd4f_19e8_4ed6_a218_188e275ecc05 = New-PSCredential -User apple -Password '#<Sensitive [value redacted]>'
$InvokeParams = @{
Name = 'PSRepository';
Method = 'set';
Property = @{
name = 'psGallery';
installationpolicy = 'Trusted';
psdscrunascredential = $febabd4f_19e8_4ed6_a218_188e275ecc05
};
ModuleName = @{
ModuleName = 'C:/pathToPuppetModules/powershellget/lib/puppet_x/powershellget/dsc_resources/PowerShellGet/PowerShellGet.psd1';
RequiredVersion = '2.2.5'
}
}

Note that it is building a credential object from plain text before passing that object to DSC; the debug logs will strip the password from being returned.

In this case, you will need some of the custom helper code that we snipped for brevity earlier, namely this function:

function new-pscredential {
[CmdletBinding()]
param (
[parameter(Mandatory = $true,
ValueFromPipelineByPropertyName = $true)]
[string]
$user,
[parameter(Mandatory = $true,
ValueFromPipelineByPropertyName = $true)]
[string]
$password
)
$secpasswd = ConvertTo-SecureString $password -AsPlainText -Force
$credentials = New-Object System.Management.Automation.PSCredential ($user, $secpasswd)
return $credentials
}

Make sure to copy that code into your terminal before you try to copy in the invocation variables and also remember to replace #<Sensitive [value redacted]> with the appropriate password string. Alternatively, you could use Get-Credential and capture the output of that command to the same variable name from your debug log.

Invoking DSC Resources with CIM Instances

Some DSC Resources use nested CIM instances to pass hashes as property values. In these cases, there’s one or more lines above the definition of the InvokeParams variable which creates these CIM instances.

$b615d113_bb6e_49e4_9776_7e895dfe20c7 = New-CimInstance -ClientOnly -ClassName 'MSFT_KeyValuePair' -Property @{'key' = 'Accept-Language' ; 'value' = 'en-US'}
$InvokeParams = @{
Name = 'xRemoteFile';
Method = 'set';
Property = @{
destinationpath = 'C:\dsc-xpsdesiredstateconfiguration-9.1.0-0-1.tar.gz';
headers = [CimInstance[]]@($b615d113_bb6e_49e4_9776_7e895dfe20c7);
uri = 'https://forge.puppet.com/v3/files/dsc-xpsdesiredstateconfiguration-9.1.0-0-1.tar.gz'
};
ModuleName = @{
ModuleName = 'C:/pathToPuppetModules/xpsdesiredstateconfiguration/lib/puppet_x/xpsdc/dsc_resources/xPSDesiredStateConfiguration/xPSDesiredStateConfiguration.psd1';
RequiredVersion = '9.1.0'
}
}

This is necessary for DSC to correctly handle the objects, especially if they have a custom CIM instance class specific to the DSC Resource being called.

Digging deeper with Pry

Sometimes, reading the debug logs doesn’t give you enough information about what’s going on. You may need to pry into the parts of Puppet that happen after the execution starts, but before the PowerShell code is executed.

When this is the case, the most powerful thing you can do is learn to interactively debug a Puppet run using a tool called pry.

Pry is a powerful tool for interactively debugging ruby. Using it with Puppet on Windows is a bit involved but it can be an extremely useful tool in your kit.

First, you’ll want to get a functional ruby environment installed separate from the PDK.

Right now, the PDK does not support prying into a Puppet run due to limitations around native gems. If you don’t know what this means right now, don’t worry too much. As you get more used to how Ruby and Puppet works it will make more sense.

You can accomplish this however you want, but make sure there is an installation of both Ruby and the associated Devkit installed when you are done. We suggest you do this via Glenn Sarti’s RubyInstaller PowerShell module:

Install-Module RubyInstaller -Scope CurrentUser
# This will kick off an interactive installation of Ruby, including the necessary dependencies
# It relies on Chocolatey (and will install it if needed)
# Strongly suggest installation option G ('2.5.1-2 (x64)') at the time of this writing.
Install-Ruby

The next step is to find the module files on disk. They’re probably in a location like this one:

C:/ProgramData/PuppetLabs/code/environments/<your environment name>/modules/

The actual location will depend on how you configured your Puppet Environments.

Next, you’re going to Set-Location to the module you want to debug, like so:

Set-Location 'C:/ProgramData/PuppetLabs/code/environments/production/modules/powershellget'

We strongly suggest opening this location in VSCode to make editing a bit easier on yourself.

In any case, you will want to delete the file Gemfile.lock in this directory and create a new file, Gemfile.local, with the following content:

gem 'fuubar'
gem 'pry-byebug'
gem 'pry-stack_explorer'

Next, you will run the following commands:

# This loads ruby 2.5 for use; if you don't do this, you won't be able to run other commands
uru 2.5

# This is only necessary after you first install a ruby version; it's a tool to manage ruby dependencies
gem install bundler

# This will cause bundler install all of the ruby dependencies into the folder you're currently in
bundle config set path '.bundle/gems'

# This will install all of your ruby dependencies
bundle install

# This will pull down a copy of the `pwslib` module for use and editing in the rest of your debugging
bundle exec rake spec_prep

At this point, you have a functioning Ruby installation and a Puppet Module setup and ready to be debugged.

The next step is to create a new file in the module folder at examples/test.pp and put your test manifest code in there.

Now, you can run your test manifest with the following command:

bundle exec puppet apply ./examples/test.pp --modulepath ./spec/fixtures/modules

If you want to run in debug mode, append --debug to the end of that command. The output you saw during a normal Puppet run will show, and the command runs until it stops at the error you are currently investigating.

Now, you’re ready to start adding pry statements to the base provider and digging around.

Prying into the base provider

For the rest of this section we’ll be looking at the dsc_base_provider.rb file in the puppetlabs-pwshlib module, which you can find at ./spec/fixtures/modules/pwshlib/puppet/provider/dsc_base_provider in the module folder. That file is the shared provider that all Puppetized DSC Resources rely on.

For this brief tutorial, we’re going to pry into invoke_get_method, but these techniques apply to all of the included methods you may want to debug.

First, find the method in the provider file (at the time of this writing, that’s around line 231):

def invoke_get_method(context, name_hash)
...
end

We’re going to add a pry binding to this method. For the purposes of this tutorial, we’re going to add it just before the DSC invocation (around line 241 at the time of this writing):

context.debug("Script:\n #{redact_secrets(script_content)}")
# This is the new code we're adding, which will tell ruby to enter a pry debugging session:
require 'pry' ; binding.pry
output = ps_manager.execute(script_content)[:stdout]

This will get you an interactive prompt inside the Puppet run:

You can call ls in that context to see what methods and variables are available:

You can type the name of one of the variables, such as name_hash and hit enter to see what it is set to:

You can use the whereami command to show where you are in the code. If you pass an integer to this command, it will show you that many lines around your current line.

Next, we’ll execute the invocation script and assign it to our own variable (this will take a bit as it’s spinning up PowerShell for the first time and running Invoke-DscResource):

In my case, it looks like we got some data for stdout, an exit code of zero, and no errors.

If we call next, pry will execute the next line of code.

Note that the marker for our current line has updated to 243. We can keep using next to walk through the code execution, investigating any variables or running any methods for ourselves to see. I’ll keep doing that til we hit line 267, where the code manipulates the output data a little bit:

Before we run these lines, lets check on some state:

Okay, so that shows us the current state of the data, the list of valid attributes, and the parameters respectively. We can then run next and check the value of data again to see what the data.select! line did:

It looks like the list of key-value pairs has been trimmed down - we do this so only the values that Puppet cares about will be evaluated in the rest of the Puppet run.

If we use next again and look at data a final time:

We see that the key for PSDscRunAsCredential has been removed, which happens because it was the only key which was in the list of parameters.

This brings us to a loop in the code. If we want to see what happens inside the loop, we need to use the step command. This command also lets you shift contexts inside of any method being called.

Note that we’re now inside the loop - if we had called next, it would’ve evaluated the whole loop and gone to the next line after it.

If we want to just let Puppet run it’s course, we can use the continue command - this will keep us in the pry session, looking for the next pry.binding in our code. Right now we only have the one, but you can have any number of them.

If instead we want to exit the puppet run entirely, we can use the exit! command, which will bring you back to your PowerShell prompt.

And that’s it for basic pry debugging! A few commands and tools to help you look around and try stuff out to investigate where you think something has gone wrong.

Places to look

Here are a few methods you may want to investigate with a pry binding and reasons for that:

canonicalize - this method retrieves the state of the DSC Resource on the node, if it exists, and loads it into a cache for comparing later - if something goes wrong here, chances are you’ll get idempotency issues.
invoke_get_method - this is responsible not just for querying the state of the DSC Resource but also for puppetizing the returned data
ps_script_content - this method builds the PowerShell script that gets invoked to query or set the state of a DSC Resource. The methods included in it, particularly prepare_credentials, prepare_cim_instances, and invoke_params are where things are most likely to go wrong when turning Puppet code into PowerShell code.

Blog: Reaching the IAC Team

Wed, 20 Jan 2021 00:00:00 +0000

{:height=“170px” width=“170px”}

To improve collaboration and avoid issues falling through the cracks, we have collected information on all the ways to get in contact with us. Whether this be to review a PR, highlight a nasty bug or really for any other reason, let us know what we did well, and what we need to do better.

Monday is Community Day

Yes, that’s right! The whole team dedicates a full day to you every week. Every Monday we go over pull requests, releases, new issues and anything else module related.

Reaching out via Slack

As part of Community Monday, we have weekly office hours on Monday 1.00pm and 5.00pm GMT alternating each week. Check the calendar for the events in your timezone. Join #office-hours on our community-slack. Between these hours the IAC team will be online and can chat about anything module related and much more!

Outside the office hours, some of us hang out in #forge-modules and #testing on our community-slack. Even if nobody from the team is around, other community members might be able to help.

Use the {:height=“50px” width=“50px”} emoji on urgent issues that require the attention of a puppet employee.

Reaching out via Jira

A link to raising an issue using Jira can be found here.

When you come across an issue with one of our supported modules and want to raise an issue, in order for to it to come directly to us there are a few things we would doing first.

Confirm this issue hasn’t already been raised by using the JIRA search function
If the repository has issues enabled, use this instead of JIRA

Once the above has been confirmed, it is recommended that you complete the following settings in the ticket. This will ensure that our team sees the ticket pretty quick.

The values we would recommend setting are:

Project: This will always need set to ‘Modules’ if you want it to land with our team.

Issue Type: In most cases you will be raising bugs against our supported modules however you can choose the description that best suits your problem.

Summary: Just a quick simple summary of the topic you are looking to highlight to us.

Component: This is used to tag the module affected and ensure the correct person sees your issue.

Description: This is one of the most important sections, add in here as much details as you possibly can, this should include:

1) Description of the issue you are seeing
2) Steps to Reproduce
3) Expected Results
4) Actual Results
5) Logs/Screenshots/Sample Manifests - in particular, logs generated with the --trace and --debug flags

For more information on writing a good bug report check out this page on RubyGarage.

Environment: What OS did did you see it on? Which puppet version is affected? What is the module version? You can even include more information like the Ruby version. Again the more information you include here, the higher likelihood of us being able to reproduce and speed up our turnaround.

Acceptance Criteria: What exactly do you want to be able to achieve with the issue being resolved?

Some things just can’t wait

If you ever find yourself in this position please don’t hesitate to reach out to our team using email and we will do our best to help you out. Our team alias is: ia_content@puppet.com.

Useful Tips

All supported modules have an examples directory with example manifests
If your struggling to write a manifest check out our tests in spec/acceptance
We have a some demos uploaded on youtube!

Other team contacts

Security Team

For security critical information use security@puppet.com.

Your alert will be handled by our security specialists and routed to the right team. See also the security page on puppet.com.

Bolt Team

Use the issues tab on the repo.

Puppet Agent Team

team-nw@puppet.com

Forge Team

forge@puppet.com

Blog: Puppetize Digital 2020 - Content Selection

Mon, 23 Nov 2020 00:00:00 +0000

Last week saw Puppetize Digital 2020, our annual conference on all things Puppet. Over the weekend, the production team uploaded all the talks to the Puppet YouTube channel, so if you missed anything, you can now catch up at your own leisure. There is a playlist with all videos, and in this post I’ll directly link to my favourites - of course focused on Puppet Content and the IAC team’s work.

Puppet Pair Programming with Visual Studio Code

The Puppet VSCode Plugin has evolved into THE canonical way to edit puppet modules. The authors of the plugin, James Pogran and Glenn Sarti are sharing the newest updates to the plugin, as well as some of the benefits of working with a modern, full-featured Development Environment:

Since last week we also provide a default gitpod and codespaces environment on our supported modules. This way you can try out and use all of that goodness directly from your browser.

The new Puppet Developer Experience: it’s all about you

Ben Ford, our Developer Advocate and Product Manager, on the state of Puppet Content.

recent updates to the Forge in form and function
announcing the Trusted Contributor programme
a demo by Daniel Carabas to show off our new public Cloud CI for modules

Puppetizing DSC

As the IAC’s headline feature this year, Michael Lombardi is talking about how we finally fixed the “DSC-update” problem. From now on, we can publish up-to-date puppet wrappers of all DSC module from the gallery (select “DSC Resource” in the filters) to the new DSC namespace on the forge. This means that any improvement on upstream DSC resources will become available in a timely manner - and without additional build steps - to all Puppet users. The way it is designed also means that from this point forward, everyone can build their local DSC resources into a puppet module using the Puppet.Dsc Powershell Module.

Automate Everything

If you need to manage a lot of module repositories, Chris Tessmer has your back. Based on a few scripts and bolt, he single-handedly (and very handily) replaces modulesync and pdksync with a much nicer alternative. puppetsync is a set of tasks and plans supporting managing SIMP’s modules on github. In his talk, Chris shows how it is used, and a number of useful tips and tricks for working with bolt plans.

Further Content Watching

ServiceNow integration
Preparing People for Proper Patching with Puppet Plans (also aspirant for alliteration award)
Roles and Profiles
Application Stack Builds
Automating Windows
- More in-depth info in the workshop
Update your code safely
Git Workflows for Puppet

You can find the entire playlist on YouTube.

Blog: Collecting GCP info and system metrics for Honeycomb

Mon, 05 Oct 2020 00:00:00 +0000

For a new and exciting project, the team is currently onboarding on Google Cloud Platform (GCP) development.

I can’t wait until I’m able to share more, but for now I can only say we’re looking at a Sinatra-hosted Ruby API server. To gain a better understanding how the service is performing, we want to collect some “traditional” metrics. Since we’re already using honeycomb for tracing tests and API calls, looking into honeycomb for more was my first choice. Most of the content here is based off the “Getting Started With Honeycomb Metrics” whitepaper at [https://www.honeycomb.io/white-papers/], made specific to Ruby and GCP.

Static runtime information

The first thing to collect is some static information about the container runtime hosting the current process. We’re using Cloud Run as our primary deployment infrastructure for the API services, so this is our first stop. The metadata server provides details about the container instance as described in “Container instance metadata server”.

The GCP ruby gems provide Google::Cloud.env to access the metadata server. Here is a pruned example of how to add metadata to all spans for easy filtering:

require 'honeycomb-beeline'
Honeycomb.configure do |config|
config.service_name = 'backend'
end
env = Google::Cloud.env
Honeycomb.client.libhoney.add_field('gcp.project_id', env.project_id)
Honeycomb.client.libhoney.add_field('gcp.instance_id', env.lookup_metadata('instance', 'id'))
Honeycomb.client.libhoney.add_field('gcp.instance_name', env.instance_name)
region_result = env.lookup_metadata "instance", "region"
region = region_result&.split('/')&.last
Honeycomb.client.libhoney.add_field('gcp.instance_region', region)
Honeycomb.client.libhoney.add_field('gcp.instance_zone', env.instance_zone)
Honeycomb.client.libhoney.add_field('gcp.instance_machine_type', env.instance_machine_type)
Honeycomb.client.libhoney.add_field('gcp.instance_blog_tags', env.instance_blog_tags)
Honeycomb.client.libhoney.add_field('gcp.knative_service_id', env.knative_service_id)
Honeycomb.client.libhoney.add_field('gcp.knative_service_revision', env.knative_service_revision)

Put this code somewhere that is loaded on app start up. For rack-based frameworks like sinatra this is the config.ru, where we include this code through require.

At the same time, we can also add private configuration flags from our service, to track their influence. For example, to capture whether FLAG_ENABLED has been set in the container’s variables:

Honeycomb.client.libhoney.add_field('flag_enabled', ENV['FLAG_ENABLED'] == 'true')

The attributes show up in the trace/span sidebar and can be used in any query:

Dynamic Data

For the actual system metrics, like CPU and memory usage, libhoney offers builders and dynamic fields (API docs). Dynamic fields get evaluated at the start of each span[1] and added to the fields sent to Honeycomb.

To access the current Resident Set Size (RSS, “memory in use”) we use the get_process_mem gem:

require 'get_process_mem'
Honeycomb.client.libhoney.add_dynamic_field('global.memory_inuse_bytes', proc { GetProcessMem.new.bytes.to_i })

The total lifetime of the process can be calculated as the difference from the start time to now:

start = Time.now
Honeycomb.client.libhoney.add_dynamic_field('global.process_uptime_seconds', proc { Time.now - start }

After a few minutes of the heartbeat API call:

Closing Notes

This post was written while implementing the first stab at collecting metrics. As we continue to improve our understanding of the capabilities and limitations of GCP, I expect we will have to revisit the exact metrics collected. For example, data written to disk in Cloud Run is stored in RAM and also counted against the service’s quota. This sounds like something to keep an eye on eventually.

Honeycomb recommends keeping system metrics in a separate dataset from application events. During development we currently have very little traffic on the service, so for simplicities sake, we keep everything global and attached to all events. We’ll have to revisit this once we get more traffic on the service.

Blog: Docker params change detection

Wed, 23 Sep 2020 00:00:00 +0000

Docker params change detection

The problem was reported by a client with the ticket MODULES-10734. After analysis, we discovered that we don’t have parameter change detection mechanisms. In this blog post, we try to explain the problem in detail with examples.

An interesting behaviour was present in docker::run component. The problem was that if any parameter was added/modified/removed puppet agent would only apply the change only if you stopped, removed the container manually and reapplied the manifest, forcing a new container creation.

The solution was to create a new function that detects if at least one parameter is changed. The detection mechanism is based on the check between parameter values from the manifest file and correspondent field from the docker inspect object of the currently running container. The solution is present in the 3.11.0 version.

So now let’s see how the problem can be reproduced by using version <= 3.10.2:

1. install puppetlabs/docker module

$: puppet module install puppetlabs/docker --version 3.10.2

2. apply this manifest:

class { 'docker': }
docker::run { 'servercore':
image => 'hello-world:linux',
restart => 'always',
net => $facts['os']['name'] ? {
'windows' => 'nat',
default => 'bridge',
},
}

3. check the image tag by running this command:

$: docker inspect --format="{{ .Config.Image }}" servercore
$: hello-world:linux

4. change the image tag from `hello-world:linux` to `hello-world:latest` and reapply the manifest.

The manifest should look like this:

class { 'docker': }
docker::run { 'servercore':
image => 'hello-world:latest',
restart => 'always',
net => $facts['os']['name'] ? {
'windows' => 'nat',
default => 'bridge',
},
}

… and now check if the change was applied

$: docker inspect --format="{{ .Config.Image }}" servercore
$: hello-world:linux

so… something is wrong, the tag was not changed to hello-world:latest. If we want to apply this change we need to do a few more steps:

stop the container: docker stop servercore
remove the container: docker rm servercore
reapply the manifest detailed above: puppet apply <manifest_file_name>

In conclusion in the puppetlabs/docker module versions <= 3.10.2 the parameter change is not detected. If we want to change some parameters for the same container, the puppet agent will not apply these changes for us until we delete the container manually. Using the latest versions(>=3.11.0) this problem is resolved by having the parameter detection mechanism implemented for the most important parameters such as image, volumes and ports. Also please take a look at the solution.

Kind regards, Adrian Iurca

Blog: DSC + Puppet: Arrived!

Mon, 21 Sep 2020 00:00:00 +0000

Earlier this year, we announced an upcoming project to drastically improve the story for using DSC Resources with Puppet. Today, we’re announcing the general availability of that project!

The Puppet.Dsc PowerShell module is now available on the PowerShell Gallery; furthermore, an initial batch of twenty-six PowerShell modules with DSC Resources have been Puppetized and published to the Forge!

We picked several of the most-downloaded modules with DSC Resources released in the last year to start with, but in the coming weeks you should find everything available on the PowerShell Gallery up on the Forge!

But today, in addition to sharing this news, I want to go into a bit more detail, both with what’s going on under the hood and with how you can use these auto-generated modules.

Installing the Modules

If you just want to get your hands on the PowerShell module to read the help and experiment on your own, you can install it like any other PowerShell module:

Install-Module -Name Puppet.Dsc
Import-Module -Name Puppet.Dsc
Get-Command -Module Puppet.Dsc
Get-Help -Name New-PuppetDscModule -ShowWindow

Some of the functions from the PowerShell module rely on the Puppet Development Kit (PDK) being installed; we suggest using Chocolatey to do so, but you can also manually download and install if you prefer.

choco install pdk

If you want to do the same sort of thing with one of the Puppetized modules from the forge, you can install them and treat them just like any other Puppet module:

# For example, installing the Puppetized ComputerManagementDsc module manually:
puppet module install dsc-computermanagementdsc --Version 8.4.0-0-0
# Or via r10k / Code Manager / Bolt:
mod 'dsc-computermanagementdsc', '8.4.0-0-0'

Puppetizing a PowerShell module

So, you want to turn a PowerShell module with DSC Resources into a Puppet module? You can do this with a single command:

# If you run this command without the PassThru flag it returns nothing to the output stream.
# Any conversion errors will be in the error stream, as usual.
# If you'd like a deeper look at what it's doing, try running it with the Verbose flag.
New-PuppetDscModule -PowerShellModuleName ComputerManagementDsc -PassThru

Unfortunately, the output from running it isn’t all that exciting:

Behind the scenes, the command:

Uses the PDK to create a new Puppet module
Searches the PowerShell Gallery for the ComputerManagementDsc module and vendor it into the new Puppet module
Converts each of the DSC Resources from the vendored PowerShell module into a Puppet Resource API type and provider
Updates the Puppet module’s metadata and documentation based on the PowerShell module.