Oracle Cloud Infrastructure Ansible Modules

Oracle Cloud Infrastructure Ansible Modules provide an easy way to create and provision resources in Oracle Cloud Infrastructure (OCI) through Ansible. These modules allow you to author Ansible playbooks that help you automate the provisioning and configuring of Oracle Cloud Infrastructure services and resources, such as Compute, Load Balancing, Database, and other Oracle Cloud Infrastructure services.

Project URL: https://github.com/oracle/oci-ansible-modules

Oracle Cloud Infrastructure Ansible Modules - Features, Architecture, Technical Details

1. Introduction

Configuration Management (CM) and Infrastructure Automation tools, such as Ansible, are widely used by DevOps professionals and system administrators to achieve “Infrastructure as Code”. These tools help system administrators

  • declare the “desired state” of their infrastructure (and optionally “actions” to reach the desired state), and have these CM tools provision the infrastructure, and configure, maintain or repair if it diverges from that “desired state”.
  • reduce risk by ensuring repeatable infrastructure provisioning and configuration
  • help achieve speed/agility by automating common deployment/configuration tasks
  • provide visibility and audit-ability of configuration management operations

This project aims to help OCI (Oracle Cloud Infrastructure) users provision, configure and manage OCI infrastructure using Ansible.

2. Overview of Ansible

Ansible is one of the leading open source tools for automation of infrastructure provisioning, configuration management and application deployment. Ansible’s simple DSL (YAML) for declaring configuration and infrastructure tasks is easy to get started and expressive. Its agent-less architecture reduces the overhead on the network by preventing the nodes from polling the controlling machine.

Ansible also has an “batteries-included” approach, and bundles a lot of Modules with it (this group of modules is informally called as the Ansible Module Index). Most of the built-in modules are contributed by the Ansible Community.

3. Objectives

  • Feature-rich implementation that DevOps professionals and system administrators can use to provision and configure OCI Services and resources
  • Wide support of OCI services - ensure that the most common OCI Services (Compute, Storage, Network, Load Balancer, Database Service) are handled. Subsequent releases would incrementally add support for features and new Services
  • Leverage the Oracle Python SDK for all OCI API interactions
  • Follow Ansible guidelines and best practices around module development - integration tests, documentation alignment etc

4. Design Goals

  • Layered on top of the OCI Python SDK. The OCI Ansible cloud modules would use the OCI Python SDK to interact with the OCI API. The OCI Ansible modules supports and honour the SDK configuration [4] when available. This ensures that the standard SDK configuration that is already configured by a user is used by OCI ansible modules.
  • Provide a uniform interface to users for functions that are common across various OCI resources - tagging resources, waiter logic to wait for resources to reach the desired state, authentication overrides, controlling idempotency

5. Overview

OCI ansible modules architecture diagram

6. Implementation Details

6.1 Supported Services

In the first version of the OCI Ansible modules, the following Services are supported:

  • Compute
  • Block Storage
  • Object Storage
  • Networking
  • Load Balancer
  • Database Service
  • Identity and Access Management

Support for the following Services are not yet implemented. They would be implemented in future releases.

  • Email Delivery
  • File Storage
  • Audit
  • DNS
  • Networking B&C scenarios
  • Container Engine for Kubernetes
  • Archive Storage
  • Registry

6.2 Prerequisites

Apart from Ansible, the OCI Ansible modules only require a properly configured installation of the OCI Python SDK to function.

6.3 Installation

We recommend (though, not require) installation of these modules and its pre-requisites in a virtual-env. The ansible modules can be installed through an installer script (the installer script ensures that the various artifacts associated with the modules (such as documentation fragments, utility libraries and the modules) are placed in the right places for the Ansible implementation to pick up).

6.4 Dynamic Inventory

Ansible can be configured to work with hosts provisioned in OCI using a static inventory file. However this approach may not work well, as OCI compute instances may come and go over time, or be created or managed by other external tools (API, console, SDK, Terraform etc). Ansible supports the inventory to be specified dynamically through a dynamic inventory script. We provide an Oracle Cloud Infrastructure(OCI) Ansible Dynamic Inventory Script in this project that users can use to ensure that the latest set of OCI compute instances are dynamically fetched and available for their playbooks to be executed upon. The dynamic inventory has a rich set of options for a user to control how information about targets are fetched, and group the fetched information along different dimensions, so that they can be used appropriately in playbooks.

6.5 Authentication

The OCI Ansible modules supports and honour the OCI SDK configuration when provided by the user. We recommend that users employ the standard OCI SDK configuration (and its profiles support) to control the authentication information to be used for executing a Play in production.

So if a user has a SDK configuration file (~/.oci/config) as described in the OCI SDK configuration documentation with their correct Tenancy and User OCIDs and API keys, when an ansible playbook that refers to OCI ansible modules is executed, the OCI ansible modules would use the authentication information configured in the SDK configuration file to authenticate the connection, and perform the action on an OCI resource.

In development and testing scenarios, if an administrator wants to override the authentication information for an individual playbook, or a play, or an ad-hoc execution, the authentication information can be overridden through the following means:

  • Ansible module options: For instance to override the Region to connect to in a single play within a playbook, the user can specify an OCI ansible module option named “region”, and the ansible module would use the value specified by “region” over the value specified in ~/.oci/config to configure the SDK to connect to OCI.
  • Environment variables: For instance to override the Region to connect to, the user can specify an environment variable named “OCI_REGION”, and the ansible module would use the value specified by “OCI_REGION” to configure the SDK to connect to OCI.

All authentication attributes that can be specified through the SDK configuration file, can be overridden using a corresponding Ansible module or an environment variable. We don’t recommend the use of Environment variables (for security reasons) and ansible module options (for security reasons and because this would make the playbook tied to a specified tenancy/user) in production.

6.6 Security and IAM Guidelines

The OCI Ansible Modules uses the authentication information specified in the standard OCI SDK configuration file while creating and configuring OCI resources.

Caution: IAM credentials referenced in the OCI SDK configuration file, grants access to resources. It is important to secure these credentials to prevent unauthorized access to Oracle Cloud Infrastructure resources. Recommendations to secure the IAM credentials in the controller node where you run ansible playbooks that uses these modules, are described in the “Credentials Management” section of the OCI IAM whitepaper at the OCI IAM documentation.

The OCI Ansible Modules allows the authentication information specified in the OCI SDK configuration file to be overridden using module options and environment variables as described in Section 6.5. Please refer to the ansible module documentation of the OCI Ansible Modules for more details. We recommend the use of OCI SDK configuration file to specify authentication information. Use the “profiles” feature in the OCI SDK configuration file to support different users. The use of environment variables and ansible module options to override Authentication information must be avoided in production scenarios. While distributing roles that use the OCI Ansible Modules, ensure that no IAM credentials are included with the roles.

6.6 Logging/Telemetry

Logging of requests to OCI can be enabled by setting log levels in the OCI Python SDK and the OCI Ansible Modules:

  • Set the log_requests variable in your ~/.oci/config to True to enable Request logging in the OCI python SDK as described in https://github.com/oracle/oci-python-sdk/blob/master/docs/logging.rst
  • Export an environment variable to enable DEBUG mode for our Ansible modules
$ export LOG_LEVEL="DEBUG"

All subsequent debug messages from an ansible playbook execution using the OCI Ansible Modules $ ansible-playbook … would go to /tmp/oci_ansible_module.log (the default logging location for our modules).

6.7 Retries/Backoff

OCI Ansible modules uses OCI Python SDK’s waiter support for jittered backoff to retry API operations that failed with connection timeouts or retryable Service errors. The modules uses the OCI Python SDK’s support for checking connection timeouts and retryable Service errors.

For all resource actions that support intermediate states before reaching the desired lifecycle state, the OCI ansible modules (by default) automatically wait until the Resource reaches the desired state before completing a play. The user can override this behaviour and choose not to wait until the desired lifecycle state is reached.

6.8 Idempotency

Ansible recommends that all the modules be idempotent, so that repeated playbook executions by users result in reaching the desired state quickly and reliably. Unless documented explicitly in the module’s ansible-doc, all resource lifecycle operations initiated through OCI Ansible Modules are idempotent. To forcefully perform a non-idempotent creation of a resource, use the force_create option.

7. Samples

A set of samples are included along with the modules to demonstrate the usage of the OCI ansible modules. These samples cover the breadth of OCI Services that the modules support. The latest list of Samples are available in the Samples directory.

Oracle Cloud Infrastructure(OCI) Ansible Dynamic Inventory Script

If you use Ansible to work with hosts provisioned in OCI, using a static inventory file may not work well, as OCI compute instances may get provisioned and terminated over time, or be created or managed by other external tools (API, console, SDK, Terraform etc). Using the OCI dynamic inventory script will help ensure that the latest set of OCI compute instances are dynamically fetched and available for your playbooks to be executed upon.

To use the OCI dynamic inventory script, grab the script and the default configuration files from

Dynamic Inventory Script

Prerequisites

Note: Before using the script, please ensure that you have a valid OCI SDK configuration. Refer OCI SDK Configuration documentation for details on how to configure ~/.oci/config.

Script and Configuration Details

The oci_inventory.py script uses the OCI Python SDK to query OCI compute instances in your tenancy, and builds a dynamic inventory that can then be used in your Ansible playbooks. Arguments to the oci_inventory.py can help you control the configuration profile to use, the compartment to limit your search to, etc.

The oci_inventory.ini configuration file can be optionally used to configure the OCI configuration profile to use, control how the inventory details are cached, and how hosts are named in your inventory.

The oci_inventory.py script accepts the following command line arguments:

usage: oci_inventory.py [-h] [--list] [--host HOST] [-config CONFIG_FILE]
              [--profile PROFILE] [--compartment COMPARTMENT]
              [--refresh-cache] [--debug]

optional arguments:
  -h, --help            show this help message and exit
  --list                List instances (default: True)
  --host HOST           Get all information about a compute instance
  -config CONFIG_FILE, --config-file CONFIG_FILE
                        OCI config file location
  --profile PROFILE     OCI config profile for connecting to OCI
  --compartment COMPARTMENT
                        Name of the compartment
  --refresh-cache, -r   Force refresh of cache by making API requests to OCI
                        (default: False - use cache files)
  --debug               Send debug messages to STDERR

The oci_inventory.py script also accepts the following environment variables:

Environment Variable Description
OCI_CONFIG_FILE Specifies the OCI SDK configuration file to use.
OCI_INI_PATH Specifies the inventory script's configuration file to use.
OCI_CONFIG_PROFILE Specifies the profile in the OCI SDK configuration file, to be used.
OCI_USER_ID Specifies the OCID of the OCI user to use to fetch the inventory.
OCI_USER_FINGERPRINT Specifies the OCI user's key-pair's fingerprint being used to use to fetch the inventory.
OCI_USER_KEY_FILE Specifies the full path including the filename of the private key of the OCI user being used to use to fetch the inventory.
OCI_TENANCY Specifies the OCID of the tenancy to use to fetch the inventory
OCI_REGION Specifies the OCI Region to use to fetch the inventory
OCI_USER_KEY_PASS_PHRASE Specifies the passphrase of the key (if encrypted), to use to fetch the inventory.
OCI_CACHE_DIR Specifies the directory where cache files of the inventory script will reside. A file named "ansible-oci.cache" will be written to this directory. It is recommended that the directory pointed to by this environment variable be read-able and write-able (unix file permissions 600) only by the user running the inventory script.
OCI_CACHE_MAX_AGE The number of seconds a cache file is considered valid. To disable caching and get the latest inventory from OCI, set this value to 0.
OCI_HOSTNAME_FORMAT Host naming format to use in the generated inventory. Use 'fqdn' to list hosts using the instance's Fully Qualified Domain Name (FQDN). Use 'public_ip' to list hosts using public IP address. Use 'private_ip' to list hosts using private IP address.

The order of precedence for the configuration used by the inventory script is:

  1. command line arguments
  2. environment variables
  3. options in script configuration file.

The configuration file used for the script defaults to ./oci_inventory.ini file. The OCI SDK configuration file defaults to ~/.oci/config file. The script uses the DEFAULT profile from the config file if no profile name is specified.

The generated inventory is grouped along the following axes:

  • region
  • compartment_name
  • availability domain
  • vcn_id
  • subnet_id
  • security_list_id
  • image_id
  • instance shape
  • freeform tags with group name as “tag_key=value”
  • defined tags with group name as “namespace#key=value”
  • metadata (key, value) with group name as “key=value”
  • extended metadata (key, value) with group name as “key=value”

By default, all non-alphanumeric characters except HASH(#), EQUALS(=), PERIOD(.) and DASH(-) in group names and host names are replaced with an UNDERSCORE(_) when the inventory is generated, so that the names can be used as Ansible groups. To disable this replacement, set sanitize_names to False in the dynamic inventory settings file(default ./oci_inventory.ini). To also replace DASH(-) when sanitize_names is True, set replace_dash_in_names to True in the settings file.

How to Use

Using a Dynamic Inventory During Playbook Execution

Ensure that you have correct OCI SDK configuration (and optionally an oci_inventory.ini). Invoke the ansible-playbook command using

$ ansible-playbook -i <path-to-inventory-file>/oci_inventory.py <your-playbook-using-the-generated-inventory>

or use the ANSIBLE_HOSTS environment variable:

$ ANSIBLE_HOSTS=<path-to-inventory-file>/oci_inventory.py ansible-playbook <your-playbook-using-the-generated-inventory>

To Disable Cache and Fetch Latest

If you are running the dynamic inventory in a standalone manner, you can use “–refresh”/”-r” to ignore the cached inventory and fetch the latest inventory from OCI:

$ <path-to-inventory-file>/oci_inventory.py --refresh

If you are using the inventory script during an ansible-playbook invocation, set the OCI_CACHE_MAX_AGE environment variable to “0”(zero) to ignore the cache, and fetch the latest inventory from OCI:

$ OCI_CACHE_MAX_AGE=0 ansible-playbook -i <path-to-inventory-file>/oci_inventory.py <your-playbook-using-the-generated-inventory>

Debugging

If you want to look at the dynamic inventory generated by the script, run it in with “–list”, and check the output.

$ <path-to-inventory-file>/oci_inventory.py --list

To print additional debug information to STDERR, use

$ <path-to-inventory-file>/oci_inventory.py --debug

Get a Single Host’s Information

The inventory script can also be configured to provide information about a single host.

$ <path-to-inventory-file>/oci_inventory.py --host <host's-ip>

The script would then return the following variables for the specified host:

{
    "availability_domain": "IwGV:US-ASHBURN-AD-1",
    "compartment_id": "ocid1.compartment.oc1..xxxxxEXAMPLExxxxx",
    "defined_tags": {},
    "display_name": "ansible-test-instance-448",
    "extended_metadata": {},
    "freeform_tags": {},
    "id": "ocid1.instance.oc1.iad.xxxxxEXAMPLExxxxx",
    "image_id": "ocid1.image.oc1.iad.xxxxxEXAMPLExxxxx",
    "ipxe_script": null,
    "launch_mode": "CUSTOM",
    "launch_options": {
      "boot_volume_type": "ISCSI",
      "firmware": "UEFI_64",
      "network_type": "VFIO",
      "remote_data_volume_type": "ISCSI"
    },
    "lifecycle_state": "AVAILABLE",
    "metadata": {
      "baz": "quux",
      "foo": "bar"
    },
    "region": "iad",
    "shape": "VM.Standard1.1",
    "source_details": {
      "image_id": "ocid1.image.oc1.iad.xxxxxEXAMPLExxxxx",
      "source_type": "image"
    },
    "time_created": "2018-01-16T12:13:35.336000+00:00"
}

FAQs

  1. The generated inventory doesn’t reflect all the compute instances in my tenancy.
  • Check if the OCI user ocid that you are specifying (either via OCI_USER or in the “profile” of your OCI SDK configuration file) has the policy permissions to list those instances. The dynamic inventory script current makes the following API operation calls. Ensure that the corresponding permissions are given to the OCI user:
    • ListCompartments
    • ListVNICAttachments
    • GetSubnet
    • GetVCN
    • GetVNIC
    • GetInstance
  • The default OCI_HOSTNAME_FORMAT is “public_ip” and so the generated inventory would only contain compute instances with a public IP. This is useful when your ansible controller node is outside the OCI VCN (as Ansible can only reach instances with public IPs). However if you are running Ansible in a compute instance within your OCI VCN that has access to all subnets within yuor VCN and can reach compute instances with private ips, set OCI_HOSTNAME_FORMAT to “private_ip” to fetch nodes with private IPs as well.

OCI Ansible Modules Frequently Asked Questions (FAQs)

1. How do I install the latest released version of the Ansible Cloud Modules?

  • Uninstall the current installed version of the modules using the uninstall.py script in the directory you had earlier cloned the OCI Ansible Cloud Modules repo to.
$ ./uninstall.py
  • Perform a git pull on your local git repo to fetch and merge all changes in the OCI Ansible Cloud Modules Github repo at https://github.com/oracle/oci-ansible-modules.
$ git pull

$ git checkout <release>
  • Install the latest release
$ ./install.py

2. How do I get web-documentation for the OCI Ansible Cloud Modules?

Web-documentation for OCI Ansible modules is available at https://oracle-cloud-infrastructure-ansible-modules.readthedocs.io

To obtain access to detailed information about using Ansible modules in the CLI, including documentation of a module’s configurable options, samples, return values, and so forth, use the ansible-doc command on the module’s name. For example, to get the documentation for the oci_bucket_facts module, execute the following command:

$ ansible-doc oci_bucket_facts

3. How do I enable debug mode for the OCI Ansible Cloud Modules?

  • Set the log_requests variable in your ~/.oci/config to True to enable Request logging in the OCI python SDK as described in https://github.com/oracle/oci-python-sdk/blob/master/docs/logging.rst
  • Export an environment variable named LOG_LEVEL with the value DEBUG to enable DEBUG mode for the OCI Ansible modules
$ export LOG_LEVEL="DEBUG"

All subsequent debug messages from an ansible playbook execution using the OCI Ansible Cloud Modules

$ ansible-playbook ....

would go to /tmp/oci_ansible_module.log (the default logging location for the OCI Ansible modules).

OCI Ansible Cloud Modules uses standard Python logging facilities for logging. To configure log messages, export an environment variable LOG_CONFIG pointing to a YAML file (as discussed in https://docs.python.org/3/howto/logging.html#configuring-logging) containing your logging configuration information Optionally, export an environment variable LOG_PATH to to your custom path where the logs must be placed.

4. In a Mac OSX controller node, I am seeing “ImportError: No module named yaml” when executing a playbook, in spite of observing that I have ansible and its requirements including PyYAML installed in my python setup.

If you are running on macOS, and you have python installed by brew, you may see a ImportError(for example: ImportError: No module named yaml). To resolve this, you must override the ansible_python_interpreter configuration option. Setting the inventory variable ansible_python_interpreter on any host will allow Ansible to auto-replace the interpreter used when executing python modules.

ansible_python_interpreter configuration option can be set in inventory file. For example:

[control-node]
localhost ansible_python_interpreter="/usr/local/Cellar/python/2.7.14_3/bin/python2.7"

OR ansible_python_interpreter configuration option can be set using -e command line option:

$ ansible-playbook sample-playbook.yml -e 'ansible_python_interpreter=/usr/local/Cellar/python/2.7.14_3/bin/python2.7'

5. ansible-doc fails with an error when using the modules through an ansible role

ansible-doc fails with a documentation error when using the Ansible Cloud Modules through the oci-ansible-modules ansible role.

$ ansible-doc oci_bucket_facts
[ERROR]: module oci_bucket_facts has a documentation error formatting or is missing documentation

The documentation fragments shipped by an ansible module that is delivered by an ansible role, is not consulted by ansible-doc. To get around this problem, use the installer script to install the OCI Ansible Cloud Modules.

6. How do I use the oci_inventory.py dynamic inventory script?

In addition to the OCI Ansible Cloud Modules, this project also provide a dynamic inventory script for OCI that you can use to construct a dynamic inventory of your OCI compute instances. For more details, see our how-to documentation at using the dynamic inventory script.

7. Any security guidelines or best practices?

  1. The OCI Ansible Cloud Modules uses the authentication information specified in the standard OCI SDK configuration file (https://oracle-cloud-infrastructure-python-sdk.readthedocs.io/en/latest/installation.html#configuring-the-sdk) while creating and configuring OCI resources.
Caution: IAM credentials referenced in the OCI SDK configuration file, grants access to resources. It is important to secure these credentials to prevent unauthorized access to Oracle Cloud Infrastructure resources. Follow the guidelines in https://docs.us-phoenix-1.oraclecloud.com/Content/Security/Reference/iam_security.htm#IAMCredentials to secure the IAM credentials in the controller node where you run ansible playbooks that uses these modules.

The OCI Ansible Cloud Modules allows the authentication information specified in the OCI SDK configuration file to be overridden using module options and environment variables. Please refer to the ansible module documentation of the OCI Ansible Cloud Modules for more details. Oracle recommends the use of OCI SDK configuration file to specify authentication information. Use the “profiles” feature in the OCI SDK configuration file to support different users. The use of environment variables and ansible module options to override Authentication information must be avoided in production scenarios. While distributing roles that use the OCI Ansible Cloud Modules, ensure that no IAM credentials are included with the roles.

  1. Logging of OCI Ansible Cloud Modules may be configured using the a file through the LOG_CONFIG environment variable, as discussed in FAQ #3 above. It is recommended that the file pointed to by LOG_CONFIG environment variable be only access-able (Unix file permissions 400 or 600) by the user running the ansible-playbook that uses OCI Ansible Cloud Modules.
  2. The OCI Ansible Dynamic Inventory Script allows you to override the directory where cache files of the inventory script will reside using the OCI_CACHE_DIR environment variable. It is recommended that the directory pointed to by OCI_CACHE_DIR environment variable be only read-able and write-able (Unix file permissions 600) by the user running the inventory script.

How to run OCI Ansible Module Tests

Unit Tests

In a virtual environment, install all pre-requisites listed in the OCI Ansible Modules Getting Started documentation page.

To run unit tests as a developer, follow these steps:

$ # install all unit test dependencies
$ pip install pytest nose mock pytest-mock
$ python -m pytest -r a --fulltrace --color yes test/units/

Cloud modules

Oracle

Note

  • (D): This marks a module as deprecated, which means a module is kept for backwards compatibility but usage is discouraged. The module documentation details page may explain more about this rationale.

OCI Ansible Modules are Copyright (c) 2018 Oracle and/or its affiliates.

This software is made available to you under the terms of the GPL 3.0 license or the Apache 2.0 license. See LICENSE.txt for more details.