Tag: database

ZooKeeper Implementation

Here we will discuss how zookeeper functions internally

Image result for The components of the ZooKeeper service
Figure 1: Components of Zookeeper service

ZooKeeper provides high availability by replicating the ZooKeeper data on each server that composes the service. We assume that servers fail by crashing, and such faulty servers may later recover. Figure 1 shows the high-level components of the ZooKeeper service. Upon receiving a request, a server prepares it for execution (request processor). If such a request requires coordination among the servers (write requests), then they use an agreement protocol (an implementation of atomic broadcast), and finally, servers commit changes to the ZooKeeper database fully replicated across all servers of the ensemble. In the case of read requests, a server simply reads the state of the local database and generates a response to the request. The replicated database is an in-memory database containing the entire data tree.

Each znode in the tree stores a maximum of 1MB of data by default, but this maximum value is a configuration parameter that can be changed in specific cases. For recoverability, we efficiently log updates to disk, and we force writes to be on the disk media before they are applied to the in-memory database. In fact, as Chubby, we keep a replay log (a write-ahead log, in our case) of committed operations and generate periodic snapshots of the in-memory database. Every ZooKeeper server services clients. Clients connect to exactly one server to submit its requests. As we noted earlier, read requests are serviced from the local replica of each server database.

Requests that change the state of the service, write requests, are processed by an agreement protocol. As part of the agreement protocol write requests are forwarded to a single server, called the leader . The rest of the ZooKeeper servers, called followers, receive message proposals consisting of state changes from the leader and agree upon state changes.

4.1 Request Processor

Since the messaging layer is atomic, we guarantee that the local replicas never diverge, although at any point in time some servers may have applied more transactions than others. Unlike the requests sent from clients, the transactions are idempotent. When the leader receives a write request, it calculates what the state of the system will be when the write is applied and transforms it into a transaction that captures this new state. The future state must be calculated because there may be outstanding transactions that have not yet been applied to the database.

For example, if a client does a conditional setData and the version number in the request matches the future version number of the znode being updated, the service generates a setDataTXN that contains the new data, the new version number, and updated time stamps. If an error occurs, such as mismatched version numbers or the znode to be updated does not exist, an errorTXN is generated instead.

4.2 Atomic Broadcast

All requests that update ZooKeeper state are forwarded to the leader. The leader executes the request and broadcasts the change to the ZooKeeper state through Zab, an atomic broadcast protocol. The server that receives the client request responds to the client when it delivers the corresponding state change. Zab uses by default simple majority quorums to decide on a proposal, so Zab and thus ZooKeeper can only work if a majority of servers are correct (i.e., with 2f + 1 server we can tolerate f failures). To achieve high throughput, ZooKeeper tries to keep the request processing pipeline full. It may have thousands of requests in different parts of the processing pipeline. Because state changes depend on the application of previous state changes, Zab provides stronger order guarantees than regular atomic broadcast.

More specifically, Zab guarantees that changes broadcast by a leader are delivered in the order they were sent and all changes from previous leaders are delivered to an established leader before it broadcasts its own changes. There are a few implementation details that simplify our implementation and give us excellent performance. We use TCP for our transport so message order is maintained by the network, which allows us to simplify our implementation. We use the leader chosen by Zab as the ZooKeeper leader, so that the same process that creates transactions also proposes them. We use the log to keep track of proposals as the write-ahead log for the in-memory database, so that we do not have to write messages twice to disk. During normal operation, Zab does deliver all messages in order and exactly once, but since Zab does not persistently record the id of every message delivered, Zab may redeliver a message during recovery. Because we use idempotent transactions, multiple delivery is acceptable as long as they are delivered in order. In fact, ZooKeeper requires Zab to redeliver at least all messages that were delivered after the start of the last snapshot.

4.3 Replicated Database

Each replica has a copy in memory of the ZooKeeper state. When a ZooKeeper server recovers from a crash, it needs to recover this internal state. Replaying all delivered messages to recover state would take prohibitively long after running the server for a while, so ZooKeeper uses periodic snapshots and only requires redelivery of messages since the start of the snapshot. We call ZooKeeper snapshots fuzzy snapshots since we do not lock the ZooKeeper state to take the snapshot; instead, we do a depth-first scan of the tree atomically reading each znode’s data and meta-data and writing them to disk.

Since the resulting fuzzy snapshot may have applied some subset of the state changes delivered during the generation of the snapshot, the result may not correspond to the state of ZooKeeper at any point in time. However, since state changes are idempotent, we can apply them twice as long as we apply the state changes in order.

For example, assume that in a ZooKeeper data tree two nodes /foo and /goo have values f1 and g1 respectively and both are at version 1 when the fuzzy snapshot begins, and the following stream of state changes arrive having the form
(transactionType, path, value, new-version)

(SetDataTXN, /foo, f2, 2)
(SetDataTXN, /goo, g2, 2)
(SetDataTXN, /foo, f3, 3)

After processing these state changes, /foo and /goo have values f3 and g2 with versions 3 and 2 respectively. However, the fuzzy snapshot may have recorded that /foo and /goo have values f3 and g1 with versions 3 and 1 respectively, which was not a valid state of the ZooKeeper data tree. If the server crashes and recovers with this snapshot and Zab redelivers the state changes, the resulting state corresponds to the state of the service before the crash.

4.4 Client-Server Interactions

When a server processes a write request, it also sends out and clears notifications relative to any watch that corresponds to that update. Servers process writes in order and do not process other writes or reads concurrently. This ensures strict succession of notifications. Note that servers handle notifications locally. Only the server that a client is connected to tracks and triggers notifications for that client. Read requests are handled locally at each server.

Each read request is processed and tagged with a zxid that corresponds to the last transaction seen by the server. This zxid defines the partial order of the read requests with respect to the write requests. By processing reads locally, we obtain excellent read performance because it is just an in-memory operation on the local server, and there is no disk activity or agreement protocol to run. This design choice is key to achieving our goal of excellent performance with read-dominant workloads.

One drawback of using fast reads is not guaranteeing precedence order for read operations. That is, a read operation may return a stale value, even though a more recent update to the same znode has been committed. Not all of our applications require precedence order, but for applications that do require it, we have implemented sync. This primitive executes asynchronously and is ordered by the leader after all pending writes to its local replica. To guarantee that a given read operation returns the latest updated value, a client calls sync followed by the read operation. The FIFO order guarantee of client operations together with the global guarantee of sync enables the result of the read operation to reflect any changes that happened before the sync was issued. In our implementation, we do not need to atomically broadcast sync as we use a leader-based algorithm, and we simply place the sync operation at the end of the queue of requests between the leader and the server executing the call to sync. In order for this to work, the follower must be sure that the leader is still the leader. If there are pending transactions that commit, then the server does not suspect the leader. If the pending queue is empty, the leader needs to issue a null transaction to commit and orders the sync after that transaction. This has the nice property that when the leader is under load, no extra broadcast traffic is generated. In our implementation, timeouts are set such that leaders realize they are not leaders before followers abandon them, so we do not issue the null transaction.

ZooKeeper servers process requests from clients in FIFO order. Responses include the zxid that the response is relative to. Even heartbeat messages during intervals of no activity include the last zxid seen by the server that the client is connected to. If the client connects to a new server, that new server ensures that its view of the ZooKeeper data is at least as recent as the view of the client by checking the last zxid of the client against its last zxid. If the client has a more recent view than the server, the server does not reestablish the session with the client until the server has caught up. The client is guaranteed to be able to find another server that has a recent view of the system since the client only sees changes that have been replicated to a majority of the ZooKeeper servers. This behavior is important to guarantee durability.

To detect client session failures, ZooKeeper uses timeouts. The leader determines that there has been a failure if no other server receives anything from a client session within the session timeout. If the client sends requests frequently enough, then there is no need to send any other message. Otherwise, the client sends heartbeat messages during periods of low activity. If the client cannot communicate with a server to send a request or heartbeat, it connects to a different ZooKeeper server to re-establish its session. To prevent the session from timing out, the ZooKeeper client library sends a heartbeat after the session has been idle for s/3 ms and switch to a new server if it has not heard from a server for 2s/3 ms, where s is the session timeout in milliseconds.

 

Postfix: error: open database /etc/postfix/generic.db: No such file or directory

Assure the precursor file ( generic NOT generic.db ) exists

ls -la /etc/postfix/generic

when your error happens typically this file does not exist … so create it :

touch /etc/postfix/generic

only then will this be of any use

 postmap /etc/postfix/generic

now you will have a populated file /etc/postfix/generic.db

How can one see the structure of a table in SQLite?

Invoke the sqlite3 utility on the database file, and use its special dot commands:
  • .tables will list tables
  • .schema [tablename] will show the CREATE statement(s) for a table or tables

There are many other useful built-in dot commands — see the documentation at http://www.sqlite.org/sqlite.html, section Special commands to sqlite3.

Example:

sqlite> entropy:~/Library/Mail>sqlite3 Envelope\ Index
SQLite version 3.6.12
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .tables
addresses              ews_folders            subjects
alarms                 feeds                  threads
associations           mailboxes              todo_notes
attachments            messages               todos
calendars              properties             todos_deleted_log
events                 recipients             todos_server_snapshot
sqlite> .schema alarms
CREATE TABLE alarms (ROWID INTEGER PRIMARY KEY AUTOINCREMENT, alarm_id,
                     todo INTEGER, flags INTEGER, offset_days INTEGER,
                     reminder_date INTEGER, time INTEGER, argument,
                     unrecognized_data BLOB);
CREATE INDEX alarm_id_index ON alarms(alarm_id);
CREATE INDEX alarm_todo_index ON alarms(todo);

Note also that SQLite saves the schema and all information about tables in the database itself, in a magic table named sqlite_master, and it’s also possible to execute normal SQL queries against that table. For example, the documentation link above shows how to derive the behavior of the .schemaand .tables commands, using normal SQL commands (see section: Querying the database schema).

Exceeding Amazon AWS Free Tier

I recently set up an Amazon Free Tier account to store some databases. However, I was stupid enough to not pay attention to the limit of 750 hours per month and created too many instances. This month I will definitely run over the 750 hours. However, my question is the following.

Once I exceeded the free tier limits in one month, would I fall out of the free tier entirely? Or would I still have the free 750 hours once the next month starts (until the end of one year from the creation of the acc)?

Solution:

As per the Hourly Usage in the Free Tier documentation, the free tier has a limit of 750 hours per month.

So in your case, you will get 750 hours back on the next month of the billing cycle.

DynamoDB find nearest integer key

I have a dynamodb table with hash as unix time, and primarykey as name

Is there a way to get the closest unix time, if it cannot find the exact one?

So if I am searching for (123343445, john), it will either return (123343445, john) or the next closest which may be (123343446, john)

I would ideally not want to check for an item then iterate up or down until I find a matching time because I do not know what the time interval is and in most cases it will not be equal.

Solution:

Here is an example in nodejs, for other languages it is similar. The code may not be perfect, but the pseudo-code is! 🙂

const params = {
    TableName: MyTableName,
    KeyConditionExpression: '#pk = :pk AND #sk >= :sk',
    Limit: 1,
    ScanIndexForward: true,
    ExpressionAttributeNames: {
      '#pk': 'name',
      '#sk': 'timestamp',
    },
    ExpressionAttributeValues: {
      ':pk': 'john', # may use some variable
      ':sk': '123343445', # may use some variable
    },
};

# /!\ Not sure about the exact syntax
new AWS.DynamoDB.DocumentClient()
  .query(params).promise()
  .then((item) => { console.log(item); });

Please note:

  • KeyConditionExpression will return all the items after the given timestamp (123343445),
  • ScanIndexForward: true will start from your timestamp
  • Limit: 1 which will return the first match only.

You may use other languages or the aws-cli, look for these parameters, they should be available.

Should you want, you may do similar queries on indexes, all you have to do is to add the IndexName: 'STRING_VALUE', in params.

WordPress configure dynamic url

Edit wp-config.php and add the following lines:
define(‘WP_HOME’, ‘/’);
define(‘WP_SITEURL’, ‘/’);

Change the values in database as:
update wp_options set option_value=”/” where option_name=”siteurl”

update wp_options set option_value=”/” where option_name=”home”

Now your siteurl IP/DNS will not be hardcoded.

Denormalized table and duplicated data

My Java project uses pure JDBC for interacting with Oracle DB (v. 12). The transaction isolation level is Read Committed.

I have a highly denormalized table, which stores an entity in set of rows. I cannot change this. Unfortunately this table has to remain this way and the reasons are independent from me.

+------+------+---------+
| date | hash | ....... |
+------+------+---------+
| date | xyz  | ....... |
| date | xyz  | ....... |
| date | xyz  | ....... |

I have two columns identifying an entity – a date and a hash. Since each entity is stored as several rows, these columns are not really unique, or a primary key, but only indexed columns. Still I want to enforce a kind of “uniqueness”, meaning that only one entity exists at the time, no matter how many rows is it made of.

Such entity can be updated couple of times a day, resulting in different values, but also different number of rows.

To make all this happen, every time I update an entity, I do two or more queries within single transaction:

delete from "table" where "date" = ? and "hash" = ?
insert into "table" values (?, ?, .....)
insert into "table" ....
... -- as many inserts as needed to store whole entity

This works fine for a single instance of application. Unfortunately, I have 2 instances working simultaneously, trying to store exactly the same data at almost the same time (they are simply primary-backup instances, but backup is also persisting – this I also have no influence on).

If this was normalized table, the solution would be to use MERGE statement, but it won’t work here.

My current solution:

What I tried to do so far is to add one more column, an ID of the instance persisting, then executing INSERT statements using SELECT as a data source and putting condition to SELECTs that there must be no data for this date/hash and app ID, otherwise SELECT provides no data to insert.

I thought it would work, but apparently it does not. I still see duplicates. I think it’s because two transactions do their delete at first, still don’t see data yet to be committed by other transaction, thus perform inserts on their own. Then the “commit” is execute and boom. Both transactions insert their data.

Other approaches that I considered:

I guess also optimistic locking won’t work, because at the final version check both transactions can still consider version to not be changed, while they are actually changed by both transactions at the same time and are about to be committed this way.

I know I could switch transaction isolation to SERIALIZABLE, but it isn’t perfect either (first of all, Oracle driver will not serialize queries, but will do optimistic approach and fail with error in case of concurrent modification, I don’t like that, it’s a “programming by exception” paradigm, an anti-pattern, then the second disadvantage is performance of course).

Are there any other solutions to such a problem?

Solution:

Your requirements, as I read them are:

  • Database structure cannot change
  • Both application must update exactly the same data at the same time
  • Optimistic locking is out because it might cause errors or a performance degradation
  • Pessimistic locking is out for the same reasons as optimistic locking

It seems like the most important thing is not what data you’re changing but what data you are reading. You need a method of determining what data the users of your system (I can’t tell whether these applications are just maintaining data or also using it) should be served.

I assume what your current queries for serving data are something like:

select * from table where date = :1 and hash = :2

If you change this to the following then you’ll always select the most recent data and if there are duplicates in time you’ll pick the first application (essentially random – change to whatever ordering you want)

select *
  from ( select t.*
              , rank() over (partition by hash 
                                 order by date desc, app_id desc) as rnk
           from table t
                )
 where rnk = 1

You can maybe put this in a view?

Then, you’re able to essentially running two separate tables in a single table. You can use MERGE etc. and can change your DELETE/INSERT statements to:

merge into table o
using (select :1, :2 ... ) n
   on ( o.date = n.date
       and o.hash = n.hash
       and o.app_id = n.app_id
           )
 when matched then
      update
         set ...
 when not matched then
      insert (...

commit;

delete from table
 where date < :1 
   and hash = :2

commit;

where you’re using the same date and hash from your MERGE statement. If the DELETE fails you don’t really mind – you’re protected from selecting the wrong data because you’ve changed your SELECT queries.

Personally, I’d acknowledge that one of your requirements have to change.

If there’s any plans for adding additional applications I’d accept the performance degradation and perform updates on this table serially using a queuing mechanism.

If there’s no plans for adding additional applications take the simple approach now and start using a locking strategy (not pretty) and just handle some known errors.

How to retrieve ALL data from an sqlite database?

A couple days ago I’ve been working with an sqlite3 database and noticed that after deleting a lot of data (an entire table (DROP) to be exact) the file size did not change (the table used 5MB) at all. Not the database I’m asking about, but maybe this is relevant to my question.

I’m having a bunch of databases that I was looking through using grep -a for a specific message, I successfully found the database in question having 50MB in size, seems perfect! However, opening the database in an sqlite3 database viewer (or the sqlite3 command line tool) shows only a little over 800 entries in the messages table. The current auto increment index is above 18.000, which is the amount of entries the table should have. Going through the .db file with grep, everything I want seems to be there, but I’m guessing it’s somehow “hidden”(?).

How can I retrieve ALL the data from an sqlite database? Note that .dump <table> does also not include the messages I’m looking for. For a quick explanation on this and why data doesn’t actually get deleted/filesize won’t shrink when I delete a 5MB table I’d be thankful too.

sqlite> .dbinfo
database page size:  4096
write format:        1
read format:         1
reserved bytes:      0
file change counter: 366
database page count: 11405
freelist page count: 10372
schema cookie:       2
schema format:       4
default cache size:  0
autovacuum top root: 0
incremental vacuum:  0
text encoding:       1 (utf8)
user version:        7
application id:      0
software version:    3008005
number of tables:    3
number of indexes:   1
number of triggers:  0
number of views:     0
schema size:         737

(Note that I did not create nor ever edited this database, it’s Instagram’s direct messages database.)

Solution:

Deleted data is not immediately removed from the database file; see change sqlite file size after “DELETE FROM table”.

As shown by the freelist page count, that file is mostly empty, and running VACUUM is likely to throw away the remains of the data you’re searching for.

There is no easy way to recover deleted data (because some important management information is missing). If you are interested in only a small amount of data, try a hex editor.

Install and configure chef server on centos 7

  1. Go to this link: https://downloads.chef.io/chef-server
  2. In the on-premise section select your OS and download your package.
  3. Here I will be using centos 7. When the rpm is downloaded you can install it via the command:
    sudo rpm -Uvh
  4. Execute the following command: chef-server-ctl reconfigure
    Because the Chef server is composed of many different services that work together to create a functioning system, this step may take a few minutes to complete.
  5. Run the following command to create an administrator: 
    $ chef-server-ctl user-create USER_NAME FIRST_NAME LAST_NAME EMAIL 'PASSWORD' --filename FILE_NAME

    An RSA private key is generated automatically. This is the user’s private key and should be saved to a safe location. The --filename option will save the RSA private key to the specified absolute path.

    For example:

    $ chef-server-ctl user-create stevedanno Steve Danno steved@chef.io 'abc123' --filename /path/to/stevedanno.pem

    If you get the following error:

    Specified config file /etc/opscode/pivotal.rb does not exist

    Then check if you have RAM available. Free some memory and rerun the

    chef-server-ctl reconfigure

  6. If you get following error: /users resource does not exist

    Then check the chef logs with following command:

    chef-server-ctl tail

    You might see an error similar to the following:
    ==> /var/log/opscode/nginx/error.log <==
    2017/09/08 11:38:28 [emerg] 29689#0: bind() to 0.0.0.0:80 failed (98: Address already in use)
    2017/09/08 11:38:28 [emerg] 29689#0: bind() to 0.0.0.0:443 failed (98: Address already in use).

    Then execute the following command:
    netstat -ntlp | grep 80
    Check which service is using the port 80 and 443. Majorly it is the httpd service. Stop the httpd service. Then the chef nginx service will start automatically without any issues.

    Then execute the create-user command it will get executed without any error.

  7. Run the following command to create an organization: 
    $ chef-server-ctl org-create short_name 'full_organization_name' --association_user user_name --filename ORGANIZATION-validator.pem

    The name must begin with a lower-case letter or digit, may only contain lower-case letters, digits, hyphens, and underscores, and must be between 1 and 255 characters. For example: 4thcoffee.

    The full name must begin with a non-white space character and must be between 1 and 1023 characters. For example: 'Fourth Coffee, Inc.'.

    The --association_user option will associate the user_name with the admins security group on the Chef server.

    An RSA private key is generated automatically. This is the chef-validator key and should be saved to a safe location. The --filename option will save the RSA private key to the specified absolute path.

    For example:

    $ chef-server-ctl org-create 4thcoffee 'Fourth Coffee, Inc.' --association_user stevedanno --filename /path/to/4thcoffee-validator.pem
  8. Enable additional features of the Chef server! The packages may be downloaded directly as part of the installation process or they may be first downloaded to a local directory, and then installed.Use DownloadsThe install subcommand downloads packages from https://packages.chef.io/ by default. For systems that are not behind a firewall (and have connectivity to https://packages.chef.io/), these packages can be installed as described below.
    Feature Command
    Chef Manage

    Use Chef management console to manage data bags, attributes, run-lists, roles, environments, and cookbooks from a web user interface.

    On the Chef server, run:

    $ chef-server-ctl install chef-manage

    then:

    $ chef-server-ctl reconfigure

    and then:

    $ chef-manage-ctl reconfigure

    Note

    Starting with the Chef management console 2.3.0, the Chef MLSA must be accepted when reconfiguring the product. If the Chef MLSA has not already been accepted, the reconfigure process will prompt for a yes to accept it. Or run chef-manage-ctl reconfigure --accept-license to automatically accept the license.
  9. Open the chef-manage UI in the browser with https:///login
    Login with the credentials of the user you just created in the above step.
    On the nodes panel if you see the following error:  Error An error occurred, please try again
    Then look at the chef logs with the command chef-server-ctl tail
    The error will be majorly with the nginx. The error that I was facing was as follows:
    FAILED SocketConnector@127.0.0.1:8983: java.net.BindException: Address already in use (Bind failed). The issue was that solr service was already running on my server before installing chef. I had to stop the solr service, the issue was resolved. I was able to see the nodes panel without any error.2017-09-11 11_03_01-Mail
  10. Chef Push Jobs

    Use Chef push jobs to run jobs—an action or a command to be executed—against nodes independently of a chef-client run.

    On the Chef server, run:

    $ chef-server-ctl install opscode-push-jobs-server

    then:

    $ chef-server-ctl reconfigure

    and then:

    $ opscode-push-jobs-server-ctl reconfigure
  11. Reporting

    Use Reporting to keep track of what happens during every chef-client runs across all of the infrastructure being managed by Chef. Run Reporting with Chef management console to view reports from a web user interface.

    On the Chef server, run:

    $ chef-server-ctl install opscode-reporting

    then:

    $ chef-server-ctl reconfigure

    and then:

    $ opscode-reporting-ctl reconfigure
  12. If you want to login into postgres database of chef and see the data present in there execute the following command:
    chef-server-ctl psql
    You will get an output similar to:
    [ERROR] You must supply a service name. Valid names include: bifrost, bookshelf, oc-id, oc_erchef, oc_id, opscode-erchef, opscode_chef, push-jobs, reporting
    This is the list of databases you can login and see the data from.
    The main database where the chef data is present is opscode-erchef (previously opscode_chef). You can login to the database with the following command:
    chef-server-ctl psql opscode-erchef
    To see the tables in the database execute the following psql command:
    \dt
    This will give you a list as below:
    Schema | Name | Type | Owner
    ——–+——————————————–+——-+—————
    public | checksums | table | opscode-pgsql
    public | clients | table | opscode-pgsql
    public | containers | table | opscode-pgsql
    public | cookbook_artifact_version_checksums | table | opscode-pgsql
    public | cookbook_artifact_versions | table | opscode-pgsql
    public | cookbook_artifacts | table | opscode-pgsql
    public | cookbook_version_checksums | table | opscode-pgsql
    public | cookbook_versions | table | opscode-pgsql
    public | cookbooks | table | opscode-pgsql
    public | data_bag_items | table | opscode-pgsql
    public | data_bags | table | opscode-pgsql
    public | environments | table | opscode-pgsql
    public | groups | table | opscode-pgsql
    public | keys | table | opscode-pgsql
    public | node_policy | table | opscode-pgsql
    public | nodes | table | opscode-pgsql
    public | opc_customers | table | opscode-pgsql
    public | opc_users | table | opscode-pgsql
    public | org_migration_state | table | opscode-pgsql
    public | org_user_associations | table | opscode-pgsql
    public | org_user_invites | table | opscode-pgsql
    public | orgs | table | opscode-pgsql
    public | policies | table | opscode-pgsql
    public | policy_groups | table | opscode-pgsql
    public | policy_revisions | table | opscode-pgsql
    public | policy_revisions_policy_groups_association | table | opscode-pgsql
    public | roles | table | opscode-pgsql
    public | sandboxed_checksums | table | opscode-pgsql
    public | users | table | opscode-pgsqlYou can see the data in every table with the follwoing command:
    select * from <table-name>;
  13. If you want to see all the settings and configurations chef is using you can see the file:
    vi /etc/opscode/chef-server-running.json
  14. Find chef version:
    head -n1 /opt/opscode/version-manifest.txt
  15. Go to the chef dashboard -> Administration-> Organizations-> Starter Kit – > download starter kit.
    unzip chef-starter.zip
    Then unzip the starter kit in workstation. You can also use your chef server for this purpose. A chef-repo directory will be created.
    cd chef-repo/.chef
    cat knife.rb
    Check the configurations in knife.rb file.
  16. Install chefdk
    rpm -ivh https://packages.chef.io/files/stable/chefdk/2.2.1/el/7/chefdk-2.2.1-1.el7.x86_64.rpm
  17. Execute the following command
    knife ssl fetch 
    If you get an permission denied error then execute the command with sudo privileges.
    After execuing the above command you will get an output similar to the above:
    WARNING: Certificates from vrushabh.novalocal will be fetched and placed in your trusted_cert
    directory (/root/chef-repo/.chef/trusted_certs)……
  18. Knife is used to create cookbooks and to upload the cookbooks to chef server, upload roles, runlist etc.
    Knife allows us to communicate with our chef server.
  19. Lets bootstrap our node to chef server. For this you will need another machine which is accessible from the machine you have unzipped the chef-repo directory. Create a user in the node machine with sudo privileges, which can be ‘ssh’ed from our workstation and chef server.
    Execute the command:
    knife bootstrap 192.168.1.240 -N chef-node –ssh-user user1 –sudo
    Where ‘chef-node’ is the node name i am giving to the node. This can be any name. If -N option is not provided the default nodename will be the hostname of the node.
    ‘user1’ is the user I created on node.
    You might get an error: Your private key could not be loaded.
    cd chef-repo then execute the bootstrap command.
    You will see an output similar to the following:
    ……
    192.168.1.240 Converging 0 resources
    192.168.1.240
    192.168.1.240 Running handlers:
    192.168.1.240 Running handlers complete
    192.168.1.240 Chef Client finished, 0/0 resources updated in 16 seconds
  20. Then goto the chef dashboard and on the nodes panel page you can see the newly added node. In the attributes panel you can see the attributes belonging to the node like CPU,RAM etc.
  21. Create a project in github(ex: chef-fluency-badge). Then cd chef-repo and execute the following command:
    git init
    git add ./
    git config –global user.email “”
    git config –global user.name “”
    git commit -am “Adding chef-repo”
    git remote add origin https://github.com//chef-fluency-badge.git
    git push -u origin master
  22. Create a cookbook with the followig command:
    knife cookbook create learn for chef < 12
    chef generate cookbook for chef => 12
    Add this learn cookbook to git repo with the following commands:
    git add ./
    git commit -m “adding learn cookbook”
    git push -u origin master
  23. Lets create apache cookbook:
    cd chef-repo
    chef generate cookbook cookbooks/apache
    cd cookbooks/apache
    Edit the metadata.rb file and enter your details. Save and close.
    cd recipes
    Edit default.rb and add the following lines:
    ##

    package 'apache2' do
        package_name 'httpd'
        action :install
end

service 'apache2' do
        service_name 'httpd'
        action [:start, :enable]
end
  24. Check the ruby syntax by running the command:
    ruby -c default.rb
    foodcritic default.rb 

    foodcritic should not give any error message.
    Now we want to change the default apache webpage.
    In the same recipes directory create websites.rb with following content:
    ##

    file "default www" do
        path '/var/www/html/index.html'
        content 'Hello world'
end
&lt;span 				data-mce-type="bookmark" 				id="mce_SELREST_start" 				data-mce-style="overflow:hidden;line-height:0" 				style="overflow:hidden;line-height:0" 			&gt;&amp;#65279;&lt;/span&gt;

    Check it with foodcritic. You can also comeback to cookbooks directory and execute:
    foodcritic apache

  25. Upload the cookbook to the chef server with the following command:
    knife cookbook upload apache
    You can also add this code to git repo.
  26. Then goto the chef-manage UI, in the policies section you will see the apache cookbook. Policy is basically configuration management for a node.
  27. In your workstation machine execute the knife node list command to get a list of nodes.
  28. Then we need to add our cookbook to run_list. Execute the following command for it:
    knife node run_list add chef-node ‘recipe[apache]’

    Then execute the following command to retreive the list of run_list:
    knife node show chef-node
    The above command gives a summary of the node like recipes, OS version etc. To get a detailed list execute the following command:
    knife node show -l chef-node

  29. Now we will dry-run our cookbook on chef-client:
    chef-client –why-run
    OR
    chef-client -W
  30. After the why-run is successful then execute the cookbook with the following command:
    chef-client

    You can check the httpd service status, it should be running. Then go to the browser and hit the <IP> of the client. It will not show our “Hello World” page but the default apache “Testing 123” page.
    This was because our websites recipe did not execute. Edit the default.rb and add the following line:
    include_recipe ‘apache::websites’

  31. Then upload the apache cookbook to chef-server:
    knife cookbook upload apache
  32. Then run chef-client your websites recipe will get executed.
  33. Suppose if we want to run the websites recipe manually everytime.
    Edit the websites.rb file and remove the include_recipe line we just added above. Then again upload the cookbook to the chef-server.
  34. Then we need to add the websites recipe to the run_list otherwise after running chef-client the websites recipe will not get executed.
    You can add the recipe with the following command:
    knife node run_list add chef-node ‘recipe[apache::websites]’ -a recipe[apache]
    or you can also execute knife node run_list add ‘apache’
  35. Now you can run chef-client it will execute the websites recipe.
  36. The client configuration can be found at /etc/chef/client.rb on the client machine.