Bits and Pieces

Authentication flow with Keystone and OpenID Connect - with cURL

2024-06-12T00:00:00+03:00

Recently I was tasked to provide an example curl commands to realize the authentication process required to get an OpenStack Identity (Keystone) token for a user federated via OpenID Connect. Usually we tend to use already established libraries like keystoneauth or gophercloud for that, but since I not so long ago had to dive into OpenID Connect anyway, this posed an interesting task.

I use Mirantis OpenStack for Kubernetes (naturally :-) ), that integrates Keystone with Keycloak via OpenID Connect, OpenStack release is Caracal.

Pre-requisites

OpenStack deployed and configured to use federation via OpenID Connect.
Account in the Identity Provider allowing you to login to OpenStack.
curl and jq installed and available.

Required access info

Obtain required access info. The commands below expect the following environment variables to be available.

OS_AUTH_URL: the OpenStack Identity endpoint address. Note that examples below expect a versioned endpoint (ends with /v3), adapt URLs accordingly if yours is not versioned.
OS_DISCOVERY_ENDPOINT: the URL of OpenID configuration discovery of the Identity Provider configured in Keystone, usually ends with .../.well-known/openid-configuration.
OS_IDENTITY_PROVIDER: Name of the identity provider configured in Keystone.
OS_PROTOCOL: Name of the protocol configured for this Identity Provider in Keystone.
OS_CLIENT_ID: The client name to use when contacting Identity Provider. Note - this is NOT your OpenStack login.
OS_CLIENT_SECRET: The password to use for the Identity Provider client. Usually it must be kept secret, as in standard Web SSO application of OpenID Connect, but here, as with Kubernetes + OpenID Connect, the CLI client is required to know it. Some Identity Providers allow to create clients without secrets (for example Keycloak does allow that).
OS_OPENID_SCOPE: The scope to use for authentication, governs how much info about you the Identity Provider will pass to the Service Provider - in this case you :-) At least (and usually only) must be openid.
OS_USERNAME: Your username with Identity Provider you want to login to OpenStack with.
OS_PASSWORD: Your password for your account with the Identity Provider.
OS_PROJECT_NAME: The name of the OpenStack project you want to authenticate to.
OS_PROJECT_DOMAIN_ID: ID of the OpenStack Identity domain that project belongs to.

Get it from Horizon

The easiest way to obtain those is to login via federation to Horizon, and download the RC file from the UI:

source this file into the active shell, it will ask you for your password

source <project>-openrc.sh

In what follows, I am assuming you've done that already, so all the necessary environment variables are present.

Authentication Flow

Now let's start with curl-ing. I use -sk to silence the progress indicator and, since I use development toy environment and self-signed TLS certificates, ignore TLS verification failures.

I am following the calls that would've been made when using keystoneauth library with v3oidcpassword authentication type.

Get OIDC token endpoint

Using the discovery endpoint, find the URL of token endpoint:

token_endpoint=$(curl -sk -X GET $OS_DISCOVERY_ENDPOINT | jq -r .token_endpoint)

Get the OIDC access token

access_token=$(curl -sk \
    -X POST $token_endpoint \
    -u $OS_CLIENT_ID:$OS_CLIENT_SECRET \
    -d "username=${OS_USERNAME}&password=${OS_PASSWORD}&scope=${OS_OPENID_SCOPE}&grant_type=password" \
    -H "Content-Type: application/x-www-form-urlencoded" \
    | jq -r .access_token)

The trick is the Content-Type - as per the OpenID Connect RFC this is how it must be done with Form Serialization, not JSON. The client id and client secret are used for HTTP Basic Authentication, again, as per that RFC.

Get the unscoped Keystone token

Now the OpenStack part. In OpenStack, tokens are issued and valid in various "scopes" - project, domain, system or unscoped.

With federation, API user is expected to first exchange the Identity Provider token for unscoped OpenStack Identity token:

unscoped_token=$(curl -sik \
    -I \
    -X POST $OS_AUTH_URL/OS-FEDERATION/identity_providers/${OS_IDENTITY_PROVIDER}/protocols/${OS_PROTOCOL}/auth \
    -H "Authorization: Bearer $access_token" \
    | grep x-subject-token \
    | awk '{print $2}' \
    | tr -d '\r')

(the result of grep + awk has new line in the end, so need to trim that out to put that value properly into JSON later).

Here already we have some inconveniences with Bash: the token arrives in the header, but the response body (in JSON) also has some info. We are ignoring it, but it may be quite useful in some applications.

Discover authentication scopes (optional)

If you do not have the intended scope of authentication at hand - project or domain or system - you can now discover the available to you scopes by making the following requests with unscoped token:

curl -sik $OS_AUTH_URL/auth/projects -H "X-Auth-Token: $unscoped_token" | jq .projects
curl -sik $OS_AUTH_URL/auth/domains -H "X-Auth-Token: $unscoped_token" | jq .domains
curl -sik $OS_AUTH_URL/auth/system -H "X-Auth-Token: $unscoped_token" | jq .system

Prepare JSON for scoped token request

Generating JSON in Bash is very awkward due to double-quotes and lots of escaping... just save the request JSON body to file (obviously not secure):

token_request=$(mktemp)
cat > $token_request << EOJSON
{
  "auth": {
    "identity": {
      "methods": [
        "token"
      ],
      "token": {
        "id": "$unscoped_token"
      }
    },
    "scope": {
      "project": {
        "domain": {
          "id": "$OS_PROJECT_DOMAIN_ID"
        },
        "name": "$OS_PROJECT_NAME"
      }
    }
  }
}
EOJSON

Get scoped token

Biggest disadvantages here, as again, the token is in the headers, but the response body contains a lot of useful info, including auth info (UUIDs of project, domain etc, group assignments, roles if explicit), and the Identity Catalog to discover the actual URL of the service endpoints we want to acces with the received token. Again, we skip all that useful info and only fetch the token:

scoped_token=$(curl -sik \
    -X POST $OS_AUTH_URL/auth/tokens \
    -d "@$token_request" -H "Content-Type: application/json" \
    | grep x-subject-token \
    | awk '{print $2}' \
    | tr -d '\r')

Remove the temporary file with token request body (tiny security improvement):

rm $token_request

Use scoped token to make request to an OpenStack service

Here hardcoded endpoint is used, however base part of it could've been discovered from the response body of the previous request.

Get the list of available images:

curl -sk \
    -X GET https://glance.it.just.works/v2/images \
    -H "X-Auth-Token: $scoped_token" \
    | jq .images

I specifically use Glance in the example as it has no project UUID in the endpoint, but many more services will need that, so their endpoints are better to discover from the catalog that was in the body of the response when we got ourselves the scoped token in the previous step.

The same but in Python

For comparison here are examples using Python:

requests only

first, very manual example using requests only, but now with proper discovery of service endpoint:

#!/usr/bin/env python
import os
import requests

fed_auth = {
    "os_discovery_endpoint": os.getenv("OS_DISCOVERY_ENDPOINT"),
    "os_identity_provider": os.getenv("OS_IDENTITY_PROVIDER"),
    "os_protocol": os.getenv("OS_PROTOCOL"),
    "os_openid_scope": os.getenv("OS_OPENID_SCOPE"),
    "os_client_secret": os.getenv("OS_CLIENT_SECRET"),
    "os_client_id": os.getenv("OS_CLIENT_ID"),
    "os_username": os.getenv("OS_USERNAME"),
    "os_password": os.getenv("OS_PASSWORD"),
    "os_project_domain_id": os.getenv("OS_PROJECT_DOMAIN_ID"),
    "os_project_name": os.getenv("OS_PROJECT_NAME"),
    "os_auth_url": os.getenv("OS_AUTH_URL"),
    "os_region_name": os.getenv("OS_REGION_NAME"),
    "os_interface": os.getenv("OS_INTERFACE"),
    "os_insecure": True,
}

VERIFY = None

if fed_auth.get("os_cacert"):
    VERIFY = fed_auth["os_cacert"]
elif fed_auth.get("os_insecure") is True:
    VERIFY = False

# discover OIDC provider token endpoint
discovery_resp = requests.get(fed_auth["os_discovery_endpoint"], verify=VERIFY)
token_endpoint = discovery_resp.json()["token_endpoint"]

# get OIDC access token
access_req_data = "username={os_username}&password={os_password}&scope={os_openid_scope}&grant_type=password".format(**fed_auth)
access_resp = requests.post(
    token_endpoint,
    verify=VERIFY,
    headers={"Content-Type": "application/x-www-form-urlencoded"},
    data=access_req_data,
    auth=(fed_auth["os_client_id"], fed_auth["os_client_secret"]),
)
access_token = access_resp.json()["access_token"]

# Exchange OIDC access token for OpenStack Identity unscoped token
unscoped_token_resp = requests.post(
    "{os_auth_url}/OS-FEDERATION/identity_providers/{os_identity_provider}/protocols/{os_protocol}/auth".format(**fed_auth),
    headers={"Authorization": f"Bearer {access_token}"},
    verify=VERIFY,
)
unscoped_token = unscoped_token_resp.headers.get("x-subject-token")

# (optional) use unscoped token to discover possible authorizaton scopes
available_project_scopes_resp = requests.get(
    "{os_auth_url}/auth/projects".format(**fed_auth),
    verify=VERIFY,
    headers={"X-Auth-Token": unscoped_token},
)

# exchange unscoped token and scope info for scoped token
scoped_auth_req = {
    "auth": {
        "identity": {
            "methods": [
                "token"
            ],
            "token": {
                "id": unscoped_token
            }
        },
        "scope": {
            "project": {
                "domain": {
                    "id": fed_auth["os_project_domain_id"]
                },
                "name": fed_auth["os_project_name"]
            }
        }
    }
}
scoped_token_resp = requests.post(
    "{os_auth_url}/auth/tokens".format(**fed_auth),
    verify=VERIFY,
    headers={"Content-Type": "application/json"},
    json=scoped_auth_req,
)
# more info on user, its roles and groups is in the JSON body of the response
scoped_token = scoped_token_resp.headers.get("x-subject-token")

catalog = scoped_token_resp.json()["token"]["catalog"]
interface = fed_auth.get("os_interface", "public")
region = fed_auth.get("os_region_name", "RegionOne")

# discover endpoint of the Image service
image_service = [s for s in catalog if s["type"] == "image"]
if not image_service:
    raise Exception("Could not find image service in catalog")
image_service = image_service[0]
image_api = [
    e["url"] for e in image_service["endpoints"]
    if e["interface"] == interface and e["region_id"] == region
]
if not image_api:
    raise Exception("Could not find required endpoint for image service")
image_api = image_api[0].rstrip("/")
if not image_api.endswith("/v2"):
    image_api += "/v2"

# use scoped token to make request to image service endpoint
# list available images
images_resp = requests.get(
    f"{image_api}/images",
    verify=VERIFY,
    headers={"X-Auth-Token": scoped_token},
)
print(images_resp.text)

openstacksdk

and then, the example using openstacksdk - a dedicated Python API library for working with OpenStack clouds. Note that with properly set up clouds.yaml configuration file that could've been just 3 lines of code:

#!/usr/bin/env python
import os
import openstack

fed_auth = {
    "os_auth_type": "v3oidcpassword",
    "os_discovery_endpoint": os.getenv("OS_DISCOVERY_ENDPOINT"),
    "os_identity_provider": os.getenv("OS_IDENTITY_PROVIDER"),
    "os_protocol": os.getenv("OS_PROTOCOL"),
    "os_openid_scope": os.getenv("OS_OPENID_SCOPE"),
    "os_client_secret": os.getenv("OS_CLIENT_SECRET"),
    "os_client_id": os.getenv("OS_CLIENT_ID"),
    "os_username": os.getenv("OS_USERNAME"),
    "os_password": os.getenv("OS_PASSWORD"),
    "os_project_domain_id": os.getenv("OS_PROJECT_DOMAIN_ID"),
    "os_project_name": os.getenv("OS_PROJECT_NAME"),
    "os_auth_url": os.getenv("OS_AUTH_URL"),
    "os_region_name": os.getenv("OS_REGION_NAME"),
    "os_interface": os.getenv("OS_INTERFACE"),
    "os_insecure": True,
}

fed = openstack.connect(load_yaml_config=False, **fed_auth)
# if you save the auth and access info to a clouds.yaml file,
# https://docs.openstack.org/openstacksdk/latest/user/config/configuration.html#config-files
# then you don't need env variables, and all the code above can be replaced
# with single call
#
# fed = openstack.connect(cloud=<cloud name>)

print(fed.list_images())

Solving 2x2 Rubik's cube

2020-12-13T00:00:00+02:00

Base combinations

Swap FRU <-> FRD - R U R' U' (with rotation)

Repeat, and cubes are back to previous places but rotated. Repeat 6 times changes nothing, everything comes back to original places
Swap FRU <-> FLU - U R U' L' U R' U' L U (with some rotation)

The D layer stays unchanged, the BRU and BLU rotate too.

Base algorithm

Build one layer, it becomes D-layer for now, should be easy enough w/o any explicit algorithm.
Use combo 2 to place the cubes of U layer in the correct places, disregarding their rotation for now.
Turn the cube upside down, the already built D-layer becomes U-layer.
Start rotating the now D-layer using combo 1 x2 or x4 - do not worry that U-layer becomes undone.
w/o rotating the whole cube, use D or D' to place the next incorrectly rotated cube of D layer to position FRD.
Repeat step 4 and 5.
Apart from some U or D rotation it should come together eventually when rotating the last incorrect D-layer cube.

Source.

Testing pykube for OpenID Connect support

2020-05-11T00:00:00+03:00

Why

During review of my PR#49 to the pykube library to add OpenIDConnect token refresh capabilities I was asked to provide some instructions on how to setup a Kubernetes cluster with OpenIDConnect for development/testing/verification of that PR.

At work, we have a central common Keycloak instance running, which I was testing my patch against. I myself never set up something like this, so this seemed like a good opportunity to get creative and learn something new in the process :-)

UPDATE

The PR in question was merged and is available in pykybe-ng >= 20.10.0.

Also note the new home for pykube-ng at Codeberg.org.

Choosing an OpenID provider

Most available sources and tools I could find are using either Google or Azure as OpenID providers together with Kubernetes.

I don't have Azure account, and when I tried Google it seems that all previous instructions were a bit outdated. Google seems to have tightened its security around OpenID (which is understandable as "Login with Google" is quite popular and may be abused). It now requires proper registered domain for callbacks with properly signed TLS cert (not a self-signed one). Both of those are overly restrictive / impossible for a local dev env.

While searching for alternatives and more info on OpenID in general I stumbled on very nice set of articles by Okta (first part), that helped me a lot with understanding the concept and flow. So I decided to give Okta a try and set up my dev env there.

Setting up Okta as IdP

Create free account on https://developer.okta.com Free tier is IMO quite enough for personal development (currently it caps at 1k monthly active users and 5 apps/clients).
On the Okta site Dashboard note the "Site URL" (see screenshot below). This is going to be the base of the URL of the OpenID provider and will be used quite a lot throughout this guide.
Add a new application of "Web" type. OpenIDConnect supports several auth flows and AFAIU only this one supports refresh tokens that kubernetes clients rely on.
Configure new app (I called mine kube)
- Default Login redirect URI pointing to localhost is almost good enough, you just have to make sure it uses HTTPS scheme - Python's oauth2 lib is picky about that. Change all other URIs to use https as well for consistency.
- Make sure you enable Refresh Token in Grant type allowed - after all we are about to test them in the first place.
On the "General" settings of your new app there are now 3 details you going to need further:
- Login redirect URI
- Client ID
- Client secret (required by Okta, but can be optional in e.g. Keycloak)
Go to "Assignments" tab of application settings and check that there is at least you yourself assigned as user allowed to use this app (should be by default but better to double check). As you see on screenshot I've added couple more dummy users to my org to play with.

Installing and configuring microk8s

I am using microk8s installed on my local Ubuntu machine for testing.

sudo snap install microk8s

I use it with all plugins disabled, including RBAC one to simplify things.

Edit arguments of kube-apiserver to add the following flags:

--oidc-username-claim=email
--oidc-issuer-url=<okta-site-url>/oauth2/default
--oidc-client-id=<client-id>

where <okta-site-url> is is your site org URL (the one looking like https://dev-NNNN.okta.com) and client-id is the Client ID of the client you've created in the previous steps.

Restart kube-apiserver for changes to take effect.

Obtaining proper kube config

Now is the tricky part where we need some actual code :-)

Effectively we need to pretend we are that web application that we've set with Okta to get the access, id and refresh tokens. There are already OpenIDConnect helpers for that, but most are either public cloud specific (Google, Azure), or otherwise hardcoded to work with other OpenID providers like Keycloak.

Again, to play with Python's libs implementing the OAuth2, I've written my own small helper that theoretically should be compatible with most OpenID providers with minimal changes.

If you've been following this guide, you'd need to set some shell variables first (here and forth I assume we are running in a virtualenv):

pip install pykube-ng requests-oauthlib oauthlib
# the same as oidc-issuer-url we've set up for kube-apiserver
export OAUTH_URI=<your okta site url>/oauth2/default
# copy from Login redirect URI on General tab of our Okta applicaiton settings
export OAUTH_REDIRECT_URI=https://localhost:8080/authorization-code/callback
# copy Client ID from General tab of your Okta application settings
export OAUTH_CLIENT_ID=<your client id>
# copy Client secret from General tab of your Okta application settings
export OAUTH_CLIENT_SECRET=<your client secret>

Now run the script. It will open your default browser with the login to your Okta site prompt. After logging in copy the whole URL your browser was redirected to and tried to open but failed (as there should be nothing serving requests on https://localhost:8080) and paste it back to the script prompt at the terminal

$ python3 k8s-oidc-helper.py
Please go to "<REDUCTED>" and authorize access.
Enter the full callback URL as attempted by the browser: https://localhost:8080/authorization-code/callback?code=<REDUCTED>

The script will output the token it got in return, as well as a snippet to merge into the users section of your kubeconfig (and persist some of that info in the /tmp/kubeuser file).

Now edit your kubeconfig to add appropriate new context with the new user we've just created and the microk8s cluster we've set up and we are ready to go:

kubectl config use-context <your new context>
kubectl get ns

Testing token refresh in pukube-ng

Now how to verify pykube functionality:

instantiate pykube from this kubeconfig and context

access some kubernetes resources

for example something global like Namespaces

$ python3 -m pykube
>>> [n.name for n in Namespace.objects(api)]
['default', 'kube-node-lease', 'kube-public', 'kube-system']

delete user's id-token from the context
- alternatively you can wait for 1 hour (token expiry in Okta), but do not access the cluster with kubectl and this context during wait time as it will refresh the token for you
try to access k8s again

W/o PR#49 the client can't authorize.

With it is successfully uses refresh-token together with the rest of the info to get a new id token and persists it in the kubeconfig file.

Whenever refresh token expires itself (not sure yet of the Okta defaults for this) you will have to repeat Obtaining proper kube config part to actually login (with password etc) to get a new refresh token.

Links

Links I've found useful and interesting while digging in all this:

Performance testing of Ansible-deploy driver for OpenStack Ironic

2017-09-01T00:00:00+03:00

What and why are we testing

For more than a year me and my colleagues are developing and promoting a new deploy driver interface for OpenStack ironic service. Contrary to the standard direct or iscsi deploy interfaces that depend on ironic-python-agent service (IPA) running in the deploy ramdisk, this new ansible deploy interface is using Ansible to provision the node, with the steps to execute during provisioning or cleaning defined as Ansible playbooks/roles/tasks etc. The main advantage of this approach is greater flexibility regarding fine-tuning of the provisioning process this deploy interface allows.

This ansible deploy interface and classic ironic drivers using it are already available as part of ironic-staging-driver project [1], and we are in the process of proposing [2] to include this interface to the ironic project itself.

One of the topics raised during discussions is the performance of this new driver interface. Contrary to the direct deploy interface that offloads most of the work to be executed by IPA that runs on the node being provisioned, this new interface involves executing code on the side of ironic-conductor service, thus we were asked to show that the ansible deploy interface can cope with deploying several nodes in parallel (50 at least) without severe performance degradation on the side of ironic-conductor service/host.

Testbed

Lab setup and tests were performed by my colleague Vasyl Saienko [3], with me mostly consulting on setting up and tuning the ansible deploy interface and Ansible itself.

The lab consisted of 4 baremetal nodes with the following relevant stats:

1x ironic host: CPU: Intel(R) Xeon(R) CPU E5-2620, 32 cores, RAM: 32GB
3x enrolled into ironic: CPU: Intel(R) Xeon(R) CPU E5-2650, 32 cores RAM: 64GB

We have deployed a minimal single host ironic installation as of 7.0.2 version (Ocata release) on the ironic host node using Mirantis Cloud Platform 1.1 [4]. The Ansible-deploy interface was taken from ironic-staging-drivers project as of stable/ocata branch. The deployment also included OpenStack Identity (keystone) and Networking (neutron) services, also of Ocata release.

We used Ansible 2.3.2.0 pip-installed from PyPI.

We enrolled the 3 slave baremetal nodes in ironic, deployed them with Ubuntu Xenial image, created 100 VMs on each of them, and enrolled those to ironic using virtualbmc utility to simulate IPMI-capable hardware nodes. This effectively gave us 300 ironic nodes to test deployment on.

Overall order of the tests was the following:

pxe_ipmitool_ansible driver, deploy 50 nodes, repeat 3 times
agent_ipmitool driver, deploy 50 nodes, repeat 3 times
agent_ipmitool driver, deploy 100 nodes, repeat 3 times
pxe_ipmitool_ansible driver, deploy 100 nodes, repeat 3 times

The deploy ramdisk used for both drivers was the tinyipa image as of stable/ocata rebuilt for usage with ansible deploy interface. We provisioned nodes with standard Ubuntu Xenial cloud image.

Concurrent deployment was done via trivial bash script calling ironic client commands in a for loop. For each iteration "virtual" nodes to be provisioned were chosen to be equally distributed between real baremetal nodes to decrease possible congestion.

Monitoring was performed with Cacti, with results presented and discussed below.

Lab tuning

The static image store was moved outside of ironic node
- due to physical network layout of the lab, this separate storage had better connection speed with baremetal nodes, and thus was performing better
The timeout for downloading the image [5] was increased.
- Current hard-coded value is more suitable for small cirros image used in OpenStack CI, and turned out to be not sufficient for downloading the image we were using given performance of our image store
- We plan to make such values actually configurable later
We switched Ansible to use paramiko transport instead of the default smart [6]
- We experienced weird problems with SSH timeouts both on initial connection and when executing tasks.
- There are number of bugs reported against Ansible that might be relevant, e.g. [7].
- Using paramiko for SSH may have introduced an extra performance cost compared to the native SSH binaries.
We increased the internal_poll_interval Ansible configuration setting [8] to 0.01 (from default 0.001).
- Available since Ansible 2.2, this setting was specifically introduced for better CPU-wise performance of Ansible in use-cases that do not require responsiveness of ansible-playbook process console output.

Results and discussion

Nodes per driver

This plot shows the number of nodes registered per each driver to set time frame reference for further graphs.

And we immediately see one of the troubles we stumbled upon - the dips in the second graph around 10:35 and 11:30. These graphs were plotted by Cacti periodically polling the ironic API for number of nodes - and at these points requests simply timed out. It happened for both drivers, so we tend to attribute this to the fact that ironic API was running as the eventlet sever instead of WSGI behind a more robust webserver (Note that running ironic as WSGI app was not yet officially supported in Ocata release).

Nodes by state

This plot shows the number of nodes in either "deploying/wait-callback" or "active" state.

Ironic host performance stats

Batches of 50

CPU usage
Load average
Memory usage
TCP counters

The sharp spikes in CPU utilization are well attributed to the TFTP server serving multiple concurrent requests.

We also see that using ansible deploy interface consumes more CPU (about 3% at peaks) and more RAM (about 3 GB) than agent-deploy. This is due to the task execution engine (Ansible) is being run locally on conductor instead of remotely on the node being deployed (IPA).

Nevertheless the overall time to provision all nodes and the average CPU load is nearly the same, and the toll multiple Ansible processes take on the conductor node is well within of what a real server suitable for such scaled baremetal cloud can handle.

Overall test (both 50 and 100 batches)

CPU usage
Load average
Memory usage
TCP counters

As expected, using direct driver interface scales better whith increasing the number of nodes and is close to O(1), while overhead of using ansible deploy interface scales closer to O(n) of number of nodes, especially for RAM consumption.

We tend to attribute such scaling difference to the fact that current internal architecture of ironic does not allow us to use Ansible as it was designed, with one ansible-playbook process executing the same playbook with identical input variables against several nodes. Instead, we launch separate ansible-playbook process for each node even when nodes are being provisioning with the same image and other settings, which obviously has negative impact on resources used.

This difference has to be taken into account when planning an (under)cloud ironic deployment that is going to allow usage the ansible deploy interface.

Conclusion

Overall we think that the ansible deploy interface performs and scales within acceptable limits on a quite standard server hardware.

References

[1]	http://git.openstack.org/cgit/openstack/ironic-staging-drivers/tree/ironic_staging_drivers/ansible

[2]	https://review.openstack.org/#/c/241946/

[3]	https://launchpad.net/~vsaienko

[4]	https://www.mirantis.com/software/mcp/

[5]	http://git.openstack.org/cgit/openstack/ironic-staging-drivers/tree/ironic_staging_drivers/ansible/playbooks/roles/deploy/tasks/download.yaml?h=stable/ocata#n10

[6]	http://docs.ansible.com/ansible/latest/intro_configuration.html#transport

[7]	https://github.com/ansible/ansible/issues/24035

[8]	http://docs.ansible.com/ansible/latest/intro_configuration.html#internal-poll-interval

Moving on with OpenStack

2017-03-07T00:00:00+02:00

So, you are changing jobs and moving to a next big thing, but would like to keep your community profile in OpenStack or probably will continue working on OpenStack on your new place?

Below are a couple of things that you may consider to do to keep your profile in OpenStack community accessible, and some other tips.

Accounts

Basically the main account to change is Ubuntu One - it ties to both Launchpad and Openstack.org

After each change ensure you can login with new, personal credentials.

UbuntuOne

Change you primary e-mail to non-corporate one.

Launchpad

Change you primary e-mail to non-corporate one.

OpenStack

https://www.openstack.org/profile/

Change your primary e-mail to a non-corporate one.

Change affiliation. Note, that if you will be officially working on OpenStack on your new job, your new company should have signed the corporate license agreement with OpenStack foundation. If you change your affiliation to 'Independent', you'll have to sign a new license agreement for independent contributors.

If concerned about Stackalytics metrics, AFAIK for some time already it uses affiliation from o.o (and not the e-mail domain) to track contributions per-company.

Gerrit

Upstream

https://review.openstack.org

Add your non-corporate e-mail, make it a primary one.

Other downstream Gerrits

Depending on whether there are open repos at all, might consider trying to add your personal e-mail to the registered e-mail addresses.

GitHub

Ensure your primary e-mail is a non-corporate one.

Git

Don't forget to change your user.email in the .gitconfig accordingly.

IRC/IRCCloud

If you were using that, change your e-mail to non-corporate one.

Code

If you have any running envs (e.g.DevStack) with upstream code - push it to OpenStack's gerrit or to the forks on your personal GitHub/BitBucket etc.

If you have any running envs with downstream code - push it to appropriate downstream repo/Gerrit.

Others

Mail

Consider exporting mail filters, usually people end up with quite elaborate setups to sort the usual mail flood.

Docs

GDrive/Confluence/Dropbox etc - whatever corporate tool you were using.

Export personal notes etc. Beware of sensitive nature of some documents.

Graphical VNC console in ironic - early prototype

2016-12-12T00:00:00+02:00

In integrated case (working as part of OpenStack deployment) nova instances deployed on ironic baremetal nodes have certain limitations compared to standard virtual machines created by nova. In particular it is not currently possible to access the graphical console of such instance for example via horizon Dashboard.

This is about to change with ironic community starting to work [1] on introducing a framework for graphical console access for baremetal nodes. As each hardware vendor implements a different way to provide graphical console access, the framework is planned to be quite generic, leaving details of actual graphical console configuration and management to a proposed GraphicalConsole interface of an ironic driver.

One interesting hardware to consider in this regard is Dell servers supporting iDRACv7 or newer [2]. The iDRAC firmware on such servers supports native access to the server’s graphical console over OpenVNC-compatible protocol directly, without a need of proprietary VNC proxies or clients. An administrator who has appropriate access to the iDRAC configuration can enable this built-in VNC server and set the password, connection timeout and SSL encryption options.

In order to test the VNC capabilities of such hardware I have implemented a prototype [3] of a graphical console interface for the DRAC driver. It uses WS-MAN HTTP API (as the rest of DRAC-specific driver interfaces) to toggle the VNC server feature on and off and set its properties. I have also created a prototype [4] of get_vnc_console method for ironic virt-driver in nova. As a result, I was able to get access to the graphical console in horizon Dashboard for the nova instance deployed on top of Dell R630 server managed by ironic.

Lessons learned

Of course no prototype is complete and without any bugs/problems discovered during testing. Here is what I’ve been hitting my head and hacking around while making this to work:

Prototype limitations

This prototype is done prior to the generic graphical console framework implementation done in ironic. Thus the prototype implementation is for now overriding the existing serial console interface in a specifically created for this purpose ironic driver. That means currently it is not possible to have both serial console and graphical console.

Conveniently though, the proposed base GraphicalConsole interface will have the same API as the current Console (SerialConsole in the future) interface. This means that once the generic framework for graphical console interfaces is implemented in ironic, this prototype can be plugged as graphical console interface basically as-is.
The interface implementation is using low-level WS-MAN Python client calls for now since support for managing iDracCardService is yet lacking from python-dracclient. The work to enable this functionality is already ongoing in the community though.
The ironic virt-driver changes are rather specific for this particular case to let me quickly test this functionality. It will be changed after the generic graphical console is implemented in Ironic and required complementary functionality is available in python-ironicclient.

iDrac VNC limitations

OpenVNC implementation in iDRAC does not seem to be complete as noVNC can not properly connect to it resulting in an apparently connected console with no graphical output [5]. A single passed encoding parameter must be disabled in noVNC code. I had to resort to noVNC patched accordingly, but the implications of such patch on access to standard VM graphical console are yet to be tested.
Password must be set on the VNC server as noVNC can not connect to it otherwise. It seems setting the password for the iDRAC VNC server to None/empty string still results in VNC server requesting a password on connection, but noVNC can not accept an empty password in its password prompt. I am not sure if this should be considered a bug in iDRAC VNC server or in noVNC.
I have not tested yet how iDRAC VNC server works with noVNC when SSL is enabled in iDRAC VNC Server.
The iDRAC VNC server is limited to a single VNC session at a time, so it is not really multi-user setup. On the other hand this still might suffice for undercloud-like use cases such as TrippleO.
Note that in the current prototype, all nodes running nova-novncproxy service (or the single one specified as vncserver_proxyclient_address in config for nova-compute with ironic virt-driver) must effectively have access to the BMC network as the built-in iDRAC VNC server is serving from its own BMC IP address. Care has to be taken to setup such proxying securely in a clustered nova deployment.

Nevertheless, this seems like an interesting and promising development on the hardware market. I consider it as yet another small step on the way forward to close the gap between baremetal and virtual servers in OpenStack and enable a unified user experience for compute service.

References

[1]	https://review.openstack.org/#/c/306074/

[2]	http://en.community.dell.com/cfs-file/__key/telligent-evolution-components-attachments/13-4491-00-00-20-44-10-34/AccessingRemoteDesktop_5F00_using_5F00_VNC_5F00_on_5F00_iDRAC.pdf

[3]	https://review.openstack.org/#/c/396661/

[4]	https://review.openstack.org/#/c/398270/

[5]	https://github.com/kanaka/noVNC/issues/712

Азы Git и Gerrit workflow в проектах OpenStack

2016-09-20T00:00:00+03:00

"git gets easier once you get the basic idea that branches are homeomorphic endofunctors mapping submanifolds of a Hilbert space" chi wai lau (@tabqwerty)

Осторожно - много англицизмов и моего личного мнения.

О чем тут вообще

Самое Главное

Не лениться и вдумчиво читать то, что git или git-review выводит в командную строку.

Там очень часто весьма внятно описано что пошло или может пойти не так, и какие есть варианты действий, с примерами команд

Git

Чумовая книжечка - Git from the Bottom Up by John Wiegley (CC BY-ND 4.0). Картинки скопированы оттуда, и общая канва повествования немного тоже.

Основные конструкции

Blob - объект хранящий “файл”. То, ради чего все затевалось. Определяется своим хэшем, который определяется только размером файла и его содержанием.
Tree - содержит ссылки на блобы и другие tree а также метадату для них. Тоже имеет хэш, определяемый хэшами содержимого и метадатой.
Commit - содержит одно дерево, и ссылки на родительские коммиты, формируя историю. Тоже имеет хэш, определяемый всем этим - содержанием дерева, коммит мессаджем и хэшами родительских коммитов.

Все остальное опирается на эти три базовых идеи.

Индекс

В отличие от (большинства?) других систем контроля версий в Git реализована двухступенчатая процедура внесения изменений - через Index (он же Staging Area).

Working area - реальные файлы на файловой системе
Index - изменения которые будут занесены в репозиторий как следующий коммит
Repository - хранилище коммитов

Бранчи и Теги

Бранчи и теги - не более чем символические ссылки на commit id имеющие человеко-читаемые имена.

HEAD - специальная ссылка на текущее (последнее чекаутное) состояние Working Tree.

<ref>^ - родитель коммита <ref>

<ref>^^ - второй родитель (есть однозначный типа MRO, но точно не помню какой). В принцие может быть больше двух родителей (^^^...) aka octopus merge, но в жизни пока не встречал/не приходилось делать.

<ref>~N - Nый родитель вдаль по истории

Основное отличие тегов в том что для них нельзя просто переопределить ссылку на коммит - надо только удалить старый тег и создать такой же новый. Но IMnsHO тому кто двигает теги надо отрубать руки.

Remotes

Ремоуты - это специальный вид бранчей, которые указывают на коммиты на удаленном репозитории (точнее в его локальной копии).

Можно - и часто нужно! - иметь много remotes в своей локальной репе.

git remote -v
git remote add <remote-name> <url>
git fetch <remote-name>

Very Basic git workflow

# создаем новый репозиторий
mkdir <repo> && cd $_ && git init
# или стягиваем готовый
git clone <remote-url> <path> && cd $_
# hack on code
git add [--patch] # заносим изменения в индекс
git commit -m “My commit” # сохраняем индекс как новый коммит
# git push [remote] # выкладываем изменения в удаленный репозиторий

Интерактивный git add --patch позволяет заносить в индекс отдельные куски изменений (hunks), а не весь файл целиком.

Mergе и конфликты

Создание коммита с несколькими родителями.

Часта ситуация когда некоторые файлы отличаются в обоих родителях, и алгоритмы Git (на самом деле весьма продвинутые) не могут однозначно определить, как должна выглядеть их “сумма”. Появляются merge conflicts которые надо чинить.

Ищем в файлах строчки

<<<[some-ref]
# код того куда мержим
====
# код того что примерживаем
>>>[other-ref]

И выбираем какая версия больше нравится. А может и переписываем кусок совсем.

git mergetool: удобная штука, которая из коробки умеет работать со многими редакторами/сравнителями (vimdiff, meld, diffuse, WinMerge, kdiff3 и т.п.). Настраивается через git-config.

Обычно в редакторе будут файлы заканчивающиеся на:

BASE - версия файла из ближайшего общего предка
LOCAL - то куда мержится
REMOTE - то что мержится

Cherry-pick

Берет дифф данного коммита и пытается сделать коммит с таким же диффом поверх другого (по сути делает копию коммита, но с другим родителем). Возможны конфликты.

Rebase

Пересаживает коммит/ветку на нового родителя. Изменяет историю!

git rebase [-i] <target-ref>

Интерактивный ребейз - очень мощная штука. Позволяет выбрать какие коммиты и в каком порядке пересаживать, слепливать несколько коммитов в один, изменять их содержимое и мессаджи.

Конфликты возможны на каждом пересаживаемом патче.

В случае мерж конфликтов при ребейзе LOCAL относится к тому на что ребейзится, а REMOTE - то что ребейзится. Это иногда не интуитивно - например, у вас был старый master, от которого отбранчевана feature-branch. Вы обновили мастер и пытаетесь ребейзнуть свой feature-branch на него. С первого взгляда интуитивно кажется что LOCAL - это ваш код (вот, же он, локальный, еще может даже никуда не выложеный), а REMOTE - это тот код из master, что вы только что выкачали (из удаленного же хранилища, да?...). Надо просто запомнить.

Правки и изменения истории

В git есть несколько команд, которые изменяют историю коммитов, как минимум с точки зрения внешнего мира (список не полный):

git commit --amend - дополняет последний коммит изменениями находящимися в индексе. Поскольку id коммита определяется его содержимым, изменяет id коммита
git rebase - так как id коммита определяется также и id его родителей, ребейз меняет id всех коммитов попавших под ребейз.

Что тут надо учитывать:

Выложить такие изменения в уже существющую удаленную ветку на remote вы уже просто так не сможете - история разъехалась
- Заставить выложить можно через git push --force
Но теперь при этом в вашей удаленной ветке тоже изменилась история. И теперь у любого, кто уже успел скачать через pull/fetch себе вашу удаленную ветку при попытке обновления возникнут проблемы - их "локальная" история разошлась с "апстримной", а это значит мерж-конфликты и прочая головная боль на пустом месте (они-то свой код не трогали..). Не надо так с людьми обходиться.

Поэтому общее правило для такого рода изменений - они только для локальных коммитов. Как только коммит в составе ветки выложен на удаленный репозиторий, откуда эту ветку мог скачать кто-то ещё - переписывайте историю этой ветки только будучи готовыми к лучам любви от этих людей.

Но что примечательно - есть основаные на git системы, очень активно использующие amend и rebase в своей работе (см Gerrit Workflow в OpenStack).

Tools

Для любителей кнопочек и менюшечек

gitk/git-gui - на лицо ужасный (Tcl/Tk), добрый внутри “дефолтный” GUI для гита
- gitk - браузер истории
- git-gui - коммиты и проч.
gitg - весьма пристойный аскетичный Гуй на Gtk
git-cola - тоже неплохо, прикольная визуализация DAG (directed acyclic graph) дерева коммитов
SourceTree - для Win/Mac, не открытый но бесплатный, от Atlassian, на Яблоке красивый
Ваш IDE - наверное то же что-то есть (PyCharm, Eclipse+PyDev…)

Для ковбоев консоли

Git :)
tig - браузер, коммитер, диффер и проч на ncurses. Пользуюсь постоянно.
Vim плагины (у меня на нем профдеформация)
- Vim-fugitive - весьма мощная штука, но пока я не очень пользуюсь, только для сложных интерактивных add. Надо переползать плотнее…
- Vim-gitgutter - помечает добавленые/удаленные/измененные строчки, и может стейжить ханки. Так же интегрируется в vim-airline и показывает общее количество незакоммиченых изменений в открытом файле.

Fun

gource - визуализация развития гит репозитория в динамике. Просто красиво :)
```
sudo apt install gource && cd <repo> && gource
```

Gerrit Workflow в OpenStack

Gerrit - система код-ревью основанная на Гите.

В cвое время отфоркался от Rietveld написанного Гуидо ван Россумом, создателем Python.

Основной принцип - содержит ченжи, внутри каждого патч-сеты. Каждый патч-сет - это отдельный бранч. Это позволяет вовсю пользоваться rebase и commit --amend, перезаписывая локальную историю и выкладывая ее на remote, что в общем случае очень сильно не рекомендуется (см Правки и изменения истории).

Специфика сообщества OpenStack (english)

Общее описание Development Workflow

http://docs.openstack.org/infra/manual/developers.html#development-workflow
Как писать годный коммит-мессадж

https://wiki.openstack.org/wiki/GitCommitMessages

git-review

В принципе c Герритом можно работать через Git напрямую, но с git-review значительно удобнее.

sudo -H pip install -U git-review

Стоит почитать man git-review.

Настраивается через git-config:

$ cat ~/.gitconfig
…
[gitreview]
    username = <my-gerrit-user-name>
    rebase = false

Basic workflow

Change-Id

git-review добавляет пост-коммит хук, который добавляет в коммит-мессадж строчку (если её ещё там не было):

Change-Id: INNNNNN…

Change-Id: независимый, Gerrit-specific хэш, по которому Геррит определяет в какой change ему добавить новую версию коммита.
Очень важно: не изменять строчку c Change-Id при обновлении патчей. Новый Change-Id => новый change на Геррите.

Работа над новым, независимым изменением (баг, фича)

# git clone … && cd <repo>
git review -s # создает новый remote по имени gerrit
git checkout master
# git pull origin master
git checkout -b <new-feature-branch>
# hack on it
git add .
git commit
# проверяем что же мы закоммитили
git log -1 && git diff HEAD^..HEAD
# прогоняем юнит и прочие тесты
# tox [-e…]
git review

Правим свой старый патч

# если ветки нет - скачиваем ее с Gerrit:
git review -d NNNNNN # review.openstack.org/#/c/NNNNNN
# создалась новая ветка review/<user_name>/<topic>
# если ветка уже есть - переключаемся на нее:
git checkout <feature-branch>
# если мерж конфликт
git checkout master
git pull origin master
git fetch gerrit
git checkout <feature-branch>
git rebase -i master
# и резолвить конфликты
# hack on it, address reviewers comments
git add .
# не создаем новый коммит!
# а добавляем изменения из индекса в последний коммит
# (и естесственно при этом меняет его commid-id)
git commit --amend
git review

Всегда коммитим обновления через аменд, перезаписывая последний коммит.

Rebase or not Rebase?

По дефолту, при выкладывании через git-review, его pre-push hook попытается сделать ребейз вашего change на ту ветку в которую вы выкладываете (по дефолту master).

Иногда это хорошо (вы забыли обновить мастер, и теперь есть мерж-конфликты - упадет сразу, не выложив).

Это поведение отменяется ключом -R.

Но чаще всего лучше делать осознанный ребейз руками - ревьюерам вашего кода проще сравнивать разные версии патч-сетов когда между ними не было ребейза.

Отдельная история - ваш change зависит от чужого, еще не вмерженого и находящегося на review. В таком случае не ребейзить чужие патчи - общее правило хорошего тона.

Поэтому мой личный алгоритм:

Отключить авторебейз по дефолту
Новый change - всегда от мастера.
Обновляю свой, независимый старый change - ребейз только если:
- еще не было ни одного ревью, или
- если merge conflict (тут уж без вариантов)
В остальных случаях - без ребейза.

Single Russian-Ukrainian keyboard layout on Ubuntu

2014-12-20T00:00:00+02:00

My most used input language is English, with second used being Russian. Occasionally though I have to input text (e.g. in search field) in German, French or Ukrainian. For Western languages Ubuntu comes with "English (international with AltGr dead keys)" available as a choice of input layout. It works out nicely as my default English layout, and those extra accented letters are easy within reach (plus some extra handy symbols are made available). On the Cyrillic languages side things are a bit more complicated. I could not find any suitable layout in Ubuntu enabled out of the box.

It turns out there is one layout included in Ubuntu, one just has to manually turn it on:

open file /usr/share/X11/xkb/rules/evdev.extras.xml, find the layout named ruu

<variant>
  <configItem>
    <name>ruu</name>
    <shortDescription>ru</shortDescription>
    <description>Russian (with Ukrainian-Belorussian layout)</description>
    <languageList><iso639Id>rus</iso639Id>
                  <iso639Id>ukr</iso639Id>
                  <iso639Id>bel</iso639Id></languageList>
  </configItem>
</variant>

copy the whole enclosing <variant> tag as shown above
open file /usr/share/X11/xkb/rules/evdev.xml

find Russian section

<layout>
  <configItem>
    <name>ru</name>

    <shortDescription>ru</shortDescription>
    <description>Russian</description>
    <languageList>
      <iso639Id>rus</iso639Id>
    </languageList>
  </configItem>
  <variantList>
    <variant>
      <configItem>
        <name>phonetic</name>
        <description>Russian (phonetic)</description>

paste there copied ruu variant alongside other available variants

<layout>
  <configItem>
    <name>ru</name>

    <shortDescription>ru</shortDescription>
    <description>Russian</description>
    <languageList>
      <iso639Id>rus</iso639Id>
    </languageList>
  </configItem>
  <variantList>
    <variant>
      <configItem>
        <name>ruu</name>
        <shortDescription>ru</shortDescription>
        <description>Russian (with Ukrainian-Belorussian layout)</description>
        <languageList><iso639Id>rus</iso639Id>
                      <iso639Id>ukr</iso639Id>
                      <iso639Id>bel</iso639Id></languageList>
      </configItem>
    </variant>
    <variant>
      <configItem>
        <name>phonetic</name>
        <description>Russian (phonetic)</description>

restart X server

Now you can choose a "Russian (with Ukrainian-Belorussian layout)" from available layouts. It is mostly a standard Russian layout (the biggest difference being position of "ё" character), plus you can input Ukrainian- or Belorussian-specific characters via AltGr (usually right Alt), plus some more extra symbols are available via AltGr, like Ukrainian Hryvnia (₴) or some math symbols (e.g. ±). Check out the "Keyboard Layout Chart" for this layout to see all available characters.

Единая русско-украинская раскладка в Ubuntu

2014-12-20T00:00:00+02:00

Так получилось, что моим основным языком ввода является английский, вторым по частоте использования русский. Тем не менее, иногда мне надо вводить текст на немецком, франзузском и украинском языках. Для западноевропейских языков в Ubuntu есть удобная раскладка "English (international with AltGr dead keys)". Пользоваться ей как основной довольно удобно, поскольку все акценты/умляуты доступны через AltGr, плюс к ним несколько других символов. Для кириллических языков ситуация несколько более сложная. Я не смог найти подходящую раскладку в Ubuntu "из коробки".

Как оказалось, раскладка-то есть, но ее надо специально руками включить:

открыть файл /usr/share/X11/xkb/rules/evdev.extras.xml, найти вариант раскладки с именем ruu

<variant>
  <configItem>
    <name>ruu</name>
    <shortDescription>ru</shortDescription>
    <description>Russian (with Ukrainian-Belorussian layout)</description>
    <languageList><iso639Id>rus</iso639Id>
                  <iso639Id>ukr</iso639Id>
                  <iso639Id>bel</iso639Id></languageList>
  </configItem>
</variant>

скопировать всю часть между тэгами <variant> как показано выше
открыть файл /usr/share/X11/xkb/rules/evdev.xml

найти секцию описывающую Русский язык

<layout>
  <configItem>
    <name>ru</name>

    <shortDescription>ru</shortDescription>
    <description>Russian</description>
    <languageList>
      <iso639Id>rus</iso639Id>
    </languageList>
  </configItem>
  <variantList>
    <variant>
      <configItem>
        <name>phonetic</name>
        <description>Russian (phonetic)</description>

вставить ранее скопированный вариант раскладки ruu рядом с другими доступными вариантами

<layout>
  <configItem>
    <name>ru</name>

    <shortDescription>ru</shortDescription>
    <description>Russian</description>
    <languageList>
      <iso639Id>rus</iso639Id>
    </languageList>
  </configItem>
  <variantList>
    <variant>
      <configItem>
        <name>ruu</name>
        <shortDescription>ru</shortDescription>
        <description>Russian (with Ukrainian-Belorussian layout)</description>
        <languageList><iso639Id>rus</iso639Id>
                      <iso639Id>ukr</iso639Id>
                      <iso639Id>bel</iso639Id></languageList>
      </configItem>
    </variant>
    <variant>
      <configItem>
        <name>phonetic</name>
        <description>Russian (phonetic)</description>

перезапустить X server

Теперь в списке доступных раскладок появляется "Russian (with Ukrainian-Belorussian layout)". В основном это стандартная русская раскладка (главное отличие - расплолжение буквы ё), но все специфично украинские и белорусские буквы доступны через AltGr, и кроме них несколько других плюшек вроде валютного значка украинской гривны (₴) или всяких математических знаков (например ±). Чтобы увидеть все доступные символы, проверьте "Keyboard Layout Chart" для этой раскладки.

How to set DevStack with Neutron

2014-04-21T00:00:00+03:00

These tips try to solve problems I stumbled upon when starting my work on OpenStack, Grizzly release at that time. I've tried to accommodate the solutions for recent DevStack (early Juno as of this writing), but some things might still be incorrect or already fixed - YMMV.

All notes are concerning Ubuntu/Debian on an Intel CPU, so for other systems some changes might be necessary.

Update: Tried DevStack on fresh Ubuntu 14.04 LTS (Trusty Thar), updated tips.

Contents

Install DevStack
Allow VM Internet connectivity
Note on unstack.sh and rebooting
Restore settings after reboot
Sharing files between your physical machine and DevStack host
Problems with receiving IPs for VMs
Nested KVM

Install DevStack

Prepare a VM for DevStack deployment
in this VM install git
```
sudo apt-get install git
```

clone DevStack repo

git clone https://github.com/openstack-dev/devstack.git

create local.conf file in the devstack folder to enable Neutron and set some passwords (choose your own if you wish)

[[local|localrc]]
DATABASE_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_TOKEN=password
SERVICE_PASSWORD=password
ADMIN_PASSWORD=password
disable_service n-net
enable_service q-svc q-agt q-dhcp q-l3 q-meta neutron

I have a repo with my DevStack customizations, you might want to check it out.

install DevStack
```
./stack.sh
```

Now you should have a working DevStack installation. You can login into Horizon, or join the running stack with ./rejoin_stack.sh command and check console outputs of all the running services for debug and error info. There are some networks and routers created by default, so you can start VMs and attach floating IPs to them.

Allow VM Internet connectivity

You might stumble on networking issues concerning outside access, like pinging or accessing Internet resources outside of your DevStack. First you need to change the Security Group settings - the "default" group created by DevStack seems to allow everything but in fact I've always had problems with it, so just create your own security group and assign your VMs to it. Then you need to configure iptables to pass through the traffic:

sudo iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

(after this launchpad question).

Note on `unstack.sh` and rebooting

As I've found the hard way, if you ever need to reboot your DevStack VM, do not run unstack.sh before that. Simply detach from screen and/or reboot as usual. The unstacking script not only stops the services run in screen, it also alters some configuration options of your system that were introduced by running stack.sh and setting up the DevStack. Mostly it concerns network bridges from what I have seen. So if you run unstack.sh, in order to have a working DevStack installation you have to run stack.sh after it. That has a drawback that all your cloud configuration (e.g. everything stored in DB tables of OpenStack and content of stack-volume-backing-file) will be reset.

Thus, the work flow looks somewhat like this:

Start/configure DevStack with stack.sh
Join the stack with rejoin_stack.sh
Make changes, restarting services affected on per-service basis via screen to put your changes in effect.
If in need to reboot the DevStack VM, shut it down the usual way, and upon restart simply run rejoin_stack.sh again, all services will be started in screen again.
Only if you need to change the OpenStack/DevStack configuration, then run unstack.sh, make changes in local.conf / wherever you have to and then run stack.sh.

Restore settings after reboot

After reboot of the DevStack VM the following will be lost:

stack volumes (this includes created storage volumes etc.)
iptables/NAT setting
bridges configurations

Here how to restore these settings or make them permanent:

Update: Right now it appears to be so involved to cleanly restore all the settings lost after reboot that a much safer and easier way is to make peace with everything in your OpenStack being lost (all the settings and things you've created inside) and just run stack.sh again (optionally with OFFLINE = True in your local.conf to keep the code exactly as before reboot).

Stack volumes

Note - with Swift enabled actual loopback device may be other than /dev/loop0

Check that volumes are created after fresh running of ./stack.sh:

sudo losetup -a
sudo pvs
sudo vgs

You should see volume group stack-volumes existing and attached to /opt/stack/data/stack-volumes-backing-file via /dev/loop0.

After reboot the attachment of the backing file to loopback device will be lost. To make it permanent add the following line to your /etc/rc.local file, before exit 0 line:

losetup /dev/loop0 /opt/stack/data/stack-volumes-backing-file

NAT settings

After this mail-list post

permanently enable IP-forwarding

sudo sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/' /etc/sysctl.conf

permanently set iptables settings: add to /etc/interfaces

post up iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Bridges

After this ghist.

Bridge br-ex has no IP after reboot. Use the following commands to set bridges according to default settings of stack.sh:

sudo ip addr flush dev br-ex
sudo ip addr add 172.24.4.1/24 dev br-ex
sudo ip link set br-ex up
sudo route add -net 10.0.0.0/24 gw 172.24.4.1

Sharing files between your physical machine and DevStack host

As I use vim as my Python IDE I personally prefer to work directly in the console of the guest DevStack instance, but if you prefer GUI IDE (like PyCharm) you might want to have access to the code on the DevStack guest right from your host. One rather straightforward possibility is sshfs, but as it is usually pretty slow, you might want to try NFS.

The following is adopted from this post by radix

First, install the nfs-kernel-server package on the host system and then edit /etc/exports to add the following line:

full_path_to_project_on_host    *(rw,async,root_squash,no_subtree_check)

Then in the DevStack guest install nfs-common and add the following line to /etc/fstab:

address_of_host:full_path_to_project_on_host    full_path_to_project_on_guest    nfs

Don't forget to mkdir full_path_to_project_on_guest in the guest. You can then reboot the DevStack guest or just mount full_path_to_project_on_guest.

Problems with receiving IPs for VMs

Some versions of VirtIO network interface seem not to fill in checksums correctly in UDP packets (something called checksum offloading), which interferes with receiving DHCP lease from neutron/nova-network when everything is running on a single host (e.g. DevStack). To fix this add the following rule to iptables:

sudo iptables -A POSTROUTING -t mangle -p udp --dport bootpc -j CHECKSUM --checksum-fill

Nested KVM

If you run DevStack as a KVM guest, ensure that your host system has nested KVM enabled - that will greatly speed up those VMs you run inside your DevStack guest (of course your CPU has to support virtualization extensions and have them enabled in BIOS).

Check that nested KVM is enabled:

cat /sys/module/kvm_intel/parameters/nested

If it's N then try to load the module with

modprobe kvm_intel nested=1

If it worked (you get Y after checking again) to make it permanent you have to add the following line to some .conf file in /etc/modprobe.d/:

options kvm-intel nested=1

Reboot and check again.

UEFI boot order on dualbooted HP EliteBook 840

2014-04-14T00:00:00+03:00

Problem

Using UEFI+SecureBoot, dual boot Windows 8.1 / Ubuntu 13.10. Boot order can be configured in BIOS per device (HDD/ODD/PXI/USB etc) but there is no way to specify boot order of systems present on HDD. UEFI boot loader finds both Windows (present as OS Manager) and Ubuntu (as ubuntu), but always uses OS Manager first. As I intend to use Ubuntu most of the time, I need a way to make booting Ubuntu the default option.

Research

According to teh Internets such problems seem to occur reasonably often, when vendors "under-implement" UEFI on their products. One commonly proposed solution is using recent versions of boot-repair Ubuntu package (a really nice piece of engineering, saved my ass couple of times in the past). What it does is it substitutes (with backup of course) the windows efi file with accordingly renamed Ubuntu efi file, thus loading GRUB automatically, and from there user can choose what system to boot. But I find such solution a bit too risky, so I want something more native.

Another program that is related is efibootmgr

sudo apt-get install efibootmgr

It has the option to modify boot order of efi files with efibootmgr -o, but at least on my particular HP notebook it has no effect, and system boots to Windows by default despite any changes in the boot order introduced by efibootmgr. Strangely though, the option to force the boot order only on next reboot (efibootmgr -n) is working alright. So, here comes the solution I figured out with the help of this Ubuntu Forums thread

Solution

My current EFI setup looks like following

shell$: sudo efibootmgr
BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0001,0002
Boot0001* ubuntu
Boot0002* Windows Boot Manager

So I 've placed the following line in /etc/rc.local (before the final exit 0 line):

/bin/efibootmgr -n 0001

This forces next default boot to be Ubuntu on every boot of Ubuntu. The dark side - after booting to Windows, this setting naturally disappears, so the next booted by default system after Windows boot is Windows again. But actually this is exactly what I want - it sort of replicates GRUB_SAVE_DEFAULT feature which I was always using before, so that every next boot by default is the same as previous boot. Installing zero delay on UEFI bootloader (there is still couple of seconds to press F9 and go into UEFI boot menu manually), and very short delay on GRUB (2 sec just for an emergency need to boot not into standard Ubuntu) gives me a system that's fast enough to boot in unattended mode.

Update

After living with this configuration for several months I realized two things:

I do not use Windows more than once a month;
Using hibernate via TuxOnIce has some problems.

The last one is due to rc.local not being processed on resume from hibernate, and unfortunately I failed to configure TuxOnIce to execute the efibootmgr properly on resume.

I again turned to BIOS settings and this time noticed that I can in fact configure "Custom" boot section and make it the first one to boot. So now I have Ubuntu booting by default, and from Grub I can start Windows if I need, and also I can safely use SAVE_DEFAULT option of Grub. For those times when Windows has to be booted directly from EFI, there is still possibility to override first boot option during POST.

Solving the Rubik's Cube

2014-04-03T00:00:00+03:00

See notation.

Cross of the first layer

Moves cube FD -> FU

F->F, D->U: F²
F->U, D->F: D R F' R' or F' M'_D F M_D

Corners of the first layer

Moves cube FDL -> FUL

F->F, D->L, L->U: L D L'
F->U, D->F, L->L: F' U' F
F->L, D->U, L->F: ( F' R' ) D² ( R F )

Second layer

Moves cube FD

F->F, D->L: ( D L D' L' ) ( D' F' D F )
F->F, D->R: ( D' R' D R ) ( D F D' F' )

Placing side cubes of third layer

Moves cube FU -> LU: (U F R ) U (R' U' F' )

Cross of the third layer

Rotating cube UR

Rotate UR and UB: ( R M_D )⁴ U ( R M_D )⁴
Rotate UR and UF: ( R M_D )⁴ U' ( R M_D )⁴
Rotate UR and UL: ( R M_D )⁴ U² ( R M_D )⁴

Placing corner cubes of third layer

Moves cubes FLU, FLR, RUB, cube ULB is not moving

Shift by one position clockwise: ( R' F' L' F ) (R F' L F )
Shift by one position counter-clockwise: ( F' L' F R' ) ( F' L F R )

Final placement

Rotate the cube FUR

[ ( R F' R' F )² ]ⁿ {U | U' | U² } [ ( R F' R' F )² ]ⁿ

n - rotates FUR by n/3 clockwise
U - rotates FUR and RUB
U' - rotates FUR and FUL
U² - rotates FUR and LUB

Figures from the solved Rubik's Cube

2014-04-02T00:00:00+03:00

See the notation.

Donkey's Bridge (chess cube of the 2nd order): M_R² M_D² M_F²
Dots: M'_D M'_R M_D M_R
Christmann's cross: R' (M_R² M_F² U² M_R² M_F² D²) R
Plummer's cross: G_F² [G_F (U² M_R² U M_R² U² M_F² D' M_F² ) ]²
Chess cube of the 3rd order (mix of Dots and Plumer's cross): [ ( M_F² D M_F² U² M_R² U' M_R² U² ) G'_F ]² G_F² (M'_R M'_D M_R M_D) G_R G'_D M_R² M_F² M_D²
Chess cube of the 6th order (mix of Donkey's Bridge and 3rd order chess cube): [ ( M_F² D M_F² U² M_R² U' M_R² U² ) G'_F ]² G_F² (M'_R M'_D M_R M_D) G_R G'_D M_R² M_D² M_F²
"6 H": D² M_R M_F² M'_R U² G_D²
"6 dashes": R² F² M_R² B² L² G_R² M_R G'_R
"6 flags": U' B² L² U M_R² U' R² F² D F B R M_D R' B' R' M_D R² M'_D R' F' B²(?) R² B² F² M_F G_F G'_D
Meson (quark-antiquark) - 2 diaginally opposite corner cubes are rotated: L² R' D R F D F' U' F D' F' R' D' R U L²
Giant meson - the same as Meson, but 2x2x2 cubes are rotated: F' U' B U² B' U R U² R' F B D F' D² F D' L' D² L B'
Giant meson with cherries - mix of Meson and Giant meson, 2x2x2 cube is rotated but the corner is not: R' U² D B' M_D B² M'_D B' U² D' R L D² U' F M_D F² M'_D F D² U L' M'_D M'_R M_D M_R
Globe - no tile has a common border with a tile of the same color: [FBLR]²LR

Rubik's Cube

2014-04-01T00:00:00+03:00

About

Quite probably the best known puzzle in the world - cube consisting of 3x3x3 matrix of colored cubes, that can be rotated per layer.

Notation used

Choose the front face - the one facing you. Then the letters denote:

Slice rotations Whole cube rotations

F: rotate the front face of the cube clock-wise
B: rotate the rear (back) face of the cube counter-clockwise (clockwise if you would make it front)
R: rotate the right face of the cube upward (clockwise if you would make it front front)
L: rotate the left face of the cube downward (i.e. also clockwise)
U: rotate the top (upper) face of the cube to the left (i.e. clockwise)
D: rotate the bottom (down) face of the cube to the right (i.e. clockwise)
M_R: rotate the middle slice of the cube upward (clockwise, as R turn)
M_D: rotate the middle slice to the right (clockwise, as D turn)
M_F: rotate the unseen middle slice clockwise (the slice between front and rear faces)
G_R: rotate the whole cube upward (clockwise, as R turn)
G_F: rotate the whole cube clockwise (as F turn)
G_D: rotate the whole cube to the right (as D turn)

The primed letter means change of direction to counterclockwise: F', B', R', L', U', D', M'_R, M'_D, M'_F, G'_R, G'_F, G'_D.