SELinux on Amazon's AWS Linux AMI

One interesting omission from Amazon's Linux AMI is SElinux and I recently had occasion to install it on a few EC2 instances. The process of installing and enabling SELinux in this environment is actually quite strait-forward, although it can require digging through quite a bit of incorrect and obsolete documentation.

The instructions below are what worked for me using the 2012.09 relase of the AMI. 2012.09 ships with kernel (3.2.30-49.59.amzn1.x86_64), but these instruction will indeed upgrade it.

The first step is to install the following packages which include SELinux and some accompanying tools.

[root@EC2]# yum install libselinux libselinux-utils libselinux-utils selinux-policy-minimum selinux-policy-mls selinux-policy-targeted policycoreutils 

Now we have to tell the kernel to enable SELinux on boot. Append the following to the kernel line in your /etc/grub.conf for your current kernel. Note that if you want to boot into permissive mode replace enforcing=1 with permissive=1.

selinux=1 security=selinux enforcing=1

In my case the resulting /etc/grub.conf looked like:

# created by imagebuilder 
default=0
timeout=1
hiddenmenu

title Amazon Linux 2012.09 (3.2.30-49.59.amzn1.x86_64)

root (hd0)
kernel /boot/vmlinuz-3.2.30-49.59.amzn1.x86_64 root=LABEL=/ console=hvc0 selinux=1 security=selinux enforcing=1
initrd /boot/initramfs-3.2.30-49.59.amzn1.x86_64.img

Now install a new kernel and build a new RAM disk. Don't worry, the options you added above will propogate to the new kernel.

[root@EC2]# yum -y update

Relabel the root filesystem

[root@EC2]# touch /.autorelabel

Now examine /etc/selinux/config and ensure the enforcement level and policy you desire are enabled. In my case I stuck with the default fully enforced targeted policy.

# This file controls the state of SELinux on the system. 

# SELINUX= can take one of these three values: 
# enforcing - SELinux security policy is enforced. 
# permissive - SELinux prints warnings instead of enforcing. 
# disabled - No SELinux policy is loaded. 
SELINUX=enforcing
# SELINUXTYPE= can take one of these two values: 
# targeted - Targeted processes are protected, 
# mls - Multi Level Security protection. 
SELINUXTYPE=targeted

Now reboot the instance

[root@EC2]# reboot

Because the root file-system was set to be relabeled rebooting will take a few minutes longer than usual.

Once the instance comes back up log in and verify your work. If everything went as planned the getenforce command will generate the following (for full enforcement).

[root@EC2]# getenforce 
Enforcing

And you're done! SELinux is installed and operating on your instance.

Fri Dec 14 2012 00:00:00 GMT+0000 (UTC)

MySQL Replication with Minimal Downtime Using R1Soft Hot Copy for Linux

At the office a while back I was experimenting with techniques to initialize MySQL replication for both InnoDB and MyISAM tables without significant downtime. The idea of locking all tables and performing a backup to ship to the slave simply takes far too long. The method that I ultimately ended up adding to my toolbelt was an adaptation of the process outlined in this article by Badan Sergiu which uses a tool from Idera called R1Soft Hot Copy for Linux.

R1Soft Hot Copy

What differentiates this process from a more standard approach is the employment of R1Soft Hot Copy. R1Soft Hot Copy is a tool that facilities the creation a snapshot of a block device. When changes to the original device occur only the differences are placed in the snapshot in a Copy-on-Write fashion (similar to VSS in Microsoft Windows). This allows an administrator to create a functional, mountable backup of an entire device almost instantly with very little effort.

Motivation and Caveats

I'm posting these instructions because I'd like some feedback not only on my adaptation, but also on the initial method. Feel free to use any of this information, but please be careful. It worked for me, but I'm not qualified to write authoritative tutorials on the subject.

Prerequisites and Requirements

I'm going to make the assumption that the reader knows how to setup MySQL replication using the methods outlined in the official documentation and that they've read the source article mentioned above.

Also keep in mind that R1Soft Hot Copy is a Linux utility making this article not directly applicable to other operating systems.

Methods

A central theme in Badan Sergiu's article was to avoid locking tables (or using lvm). The cost of not locking tables was a restart of the MySQL service itself on the master; meaning that even read queries were not able to be processed momentarily. My idea was to instead flush and lock tables in the standard fashion while creating the Hot Copy mount. That should allow read queries to still be processed and connection attempts to succeed. Writes will be temporarily blocked, but only briefly and clients should have an error free, albeit slower, experience.

Step 1: Install R1Soft Hot Copy

Use the instructions on Idera's website to install Hot Copy and then run

# hcp-setup --get-module

on the master.

Step 2: Configure master

Enable binary logging on the master server and configure a server id in my.cnf.

log-bin=mysql-bin
server-id=1

On the master create a user specifically to be used for replication.

mysql> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'SLAVE_IP_OR_HOSTNAME' IDENTIFIED BY 'slavepass';

Step 3: Create/mount a snapshot

Ensure mysql has flushed all data to disk and then lock tables so no writes can occurr.

mysql> FLUSH TABLES WITH READ LOCK;

Obtain log coordinates. Record the values of the File and Position fields.

mysql> SHOW MASTER STATUS;
+------------------+----------+--------------+------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------------+----------+--------------+------------------+
| mysql-bin.000002 | 1234     |              |                  |
+------------------+----------+--------------+------------------+

Create and mount the snapshot on the master. Because all tables are locked the coordinates obtained above will be consistent with the data in the snapshot.

# hcp -o /dev/sda2

... where /dev/sda2 is the device containing the filesystem which houses the MySQL databases to be replicated. Watch the output for the resulting mount point. This process should take mere seconds.

Release locks on the tables. This will return operation on the master to normal.

mysql> UNLOCK TABLES;

Step 4: Shutdown the slave's mysqld and copy the data

Run these commands on the slave:

# /etc/init.d/mysql stop
# rm -rf /var/lib/mysql
# rsync -avz root@MASTER_IP_OR_HOST:/var/hotcopy/sda2_hcp1/lib/mysql /var/lib/

... where /var/lib/mysql is an example path to MySQL's data.

Step 5: Unmount the snapshot on the master

# hcp -r /dev/hcp1

Step 6: Configure the slave's identity and start MySQL

Edit /etc/mysql/my.cnf on the slave and set a server id.

[mysqld]
server-id=2

# /etc/init.d/mysql start

Step 7: Configure and start slave

Now it's time to point the slave at the master and start replication. The MASTER_LOG_FILE and MASTER_LOG_POS should be set to the File and Position fields recorded in Step 3.

mysql> CHANGE MASTER TO
    ->     MASTER_HOST='MASTER_IP_OR_HOST',
    ->     MASTER_USER='repl',
    ->     MASTER_PASSWORD='slavepass',
    ->     MASTER_LOG_FILE='mysql-bin.000002',
    ->     MASTER_LOG_POS=1234;

mysql> START SLAVE;

Conclusion

At this point replication should be running and the only major service interruption was that writes were blocked for a short period on the master.

There's nothing fundamentally different in the finished product between replication setup in this fashion and a more typical dump-and-copy process. That means monitoring and maintenance should be quite standard.

Thanks

Also, thanks Badan Sergiu for posting the original article. It helped me immensely.

Sun Nov 18 2012 00:00:00 GMT+0000 (UTC)

Nubimus - A Programmatic Dialogue

Nubimus123: Hey, hey!

Pr0gram4tez: Ah, dear Nubimus. What inspires this chat window so close to happy hour on a Friday? Surely not a bug wreaking havoc on your users or offensive performance bottlenecks?

Nubimus123: No, Programates. Quite the opposite, in fact. I was interrupting your work to invite you for a beer to celebrate my team's recent success. Surely you will join me.

Pr0gram4tez: Perhaps, my friend. First tell me about your success. I wish to learn from your work while my head is still clear. My week was full of disappointment and your youthful wisdom and inspiration may be necessary for solutions in the week that follows.

Nubimus123: Gladly, Programetes. The victory was one of cryptography. Our client's data contains highly sensitive personal information and credit card numbers. They sought us out to secure their data that it never be consumed by either hacker or fool.

Pr0gram4tez: By Schneier! The protection of data of that nature is indeed important and critical to the order of the state.

Nubimus123: And that is why they rightly entrusted us with securing it.

Pr0gram4tez: Tell me, fellow developer, how did you achieve the security necessary to meet the requirements of your customer. Did you labor diligently designing an algorithm channelling all of your inventiveness? Did you employ all of your training and education designing a computation that was quick to perform while at the same time married to the highest standard of protection? Did you seek the cutting edge of mathematics and piety? My excitement implores an explanation!

Nubimus123: LOL!!!!!111 I must confess to you, who have always been honest with me. It all boils down to how we got the contract initially. Our bid was low because we allowed the gods to do the work for us. We used the cryto API in the framework crafted by the Olympians themselves.

Pr0gram4tez: How wonderful! It is said that mortals achieve their highest greatness when letting the gods handle the lofty computation so that they may focus their efforts on terrestrial business logic.

Nubimus123: Exactly.

Pr0gram4tez: Tell me, friend Nubimus, what class of divinely-crafted algorithms did you chose? Perhaps a symmetric-key cipher with Hermetic dispatch? Of either the stream or block subtype? Maybe even an asymmetric-key cipher with Alice and Bob unaware of each other's cryptographic secrets?

Nubimus123: SMH... I believe you've failed to learn the very lesson you described mere sentences ago, Programates. Such details are of the divine and, while important, could only be made poorly by simple, mortal software developers. Though I believe the default algorithm for the framework was something called Athena’s Encryption Standard or AES. I'm lead to believe it's the finest cryptographic work produced by god or man. I know not of style or tactics.

Pr0gram4tez: Interesting. The framework is so robust that the implementation required virtually no attention. You've not studied the algorithm itself?

Nubimus123: Like I said. That's a matter for the gods.

Pr0gram4tez: We can agree that the code behind the cryptography is hallowed, but I'm not aware of methods that tailor the implementation to your business case as a matter of course in any framework. There must be much to the implementation, surely. Can you not harm your client if your implementation is incorrect? Could not the gods themselves attempt to injure your business interests?

Nubimus123: Assuredly not, Programetes. The framework's code is not only holy but was developed in pairs and certified by a consortium of deities eliminating the possibility of any one Olympian's mischief or mistake.

Pr0gram4tez: The first question remains unanswered, methinks. It is not possible for you to hinder the service with even the most well-intentioned practical application?

Nubimus123: Perhaps a layman. We are a professional team. We may not be gods, but I'm confident we consumed the framework as intended. Besides, the gods will assist us. We’ve prayed to them in social media. We’ve sacrificed to them with licensing fees.

Pr0gram4tez: The admission of the potential fallibility of an implementation using the framework troubles me, even if by a neophyte. I can accept the quality of the framework as it stands but you've only offered the credentials of your team as proof the implementation is correct. You say you're professionals, but admit that you accepted the algorithm as a default without significant technical consideration. Did you investigate other algorithms offered by the framework? They’re equally divine and perhaps better suited to your toil.

Nubimus123: No, old friend. We have to deliver! Besides, the customer will be satisfied with reasonable security. What they're really paying for is our work-flow and interface. There's no way parameters to a cryptographic problem will significantly impact the effective security of our application in a negative way within the realm of practicality.

Pr0gram4tez: Whether implemented by either god or man are the cryptographic parameters not identically effective or ineffective? Speed and stability may be the domain of the gods, but would a small key size make you equally vulnerable regardless of the algorithm's origin? Can you not impart your human fallibility on these holy tools?

Nubimus123: I must admit, Programetes, I'm unaware of the practical impact of key size in our implementation, but it's certainly adjustable. It's a problem easily solved with configuration. It's still overwhelmingly in the hands of the gods who sold us the framework.

Pr0gram4tez: Very well, Nubimus. But keep in mind even with adequately-sized keys there are more cryptographic parameters such as the mode of operation. Even if adjustable at deploy-time are you aware of them now? Can you guarantee the data provided by your client will be ciphered optimally upon delivery so you don't have to trouble them for migrations later?

Nubimus123: Another admission, oh thorough Promgrametes, follows. I have no answer as I've not studied modes of operation, but can note it for consideration with the team next week. Again, likely a matter of configration. You've proven the need for an audit so our configuration is correct upon delivery, but nothing has been stated yet indicating code needs changed or that our choice was flawed.

Pr0gram4tez: Please understand my goal is not to convince you to change your code, dear Nubimus! I'm simply trying to understand how productive the framework your team uses is. Perhaps I've turned up some points that require attention as is common for a grey-beard like myself. I do have some questions that are somewhat higher-level if you have time before we leave.

Nubimus123: My thirst grows, and my brain becomes weary on this Friday afternoon, but in exchange for the service you've provided I will certainly entertain them.

Pr0gram4tez: My gratitude is immense, Nubimus. Well then, the keys... Does the framework you employ provide a secure key storage facility?

Nubimus123: It does indeed, but we don't use it. In effort to centralize our data we store the keys directly in the application's relational store.

Pr0gram4tez: I'm puzzled, Nubimus. You store the ciphertext in the relational store, correct?

Nubimus123: That is true, Progametes.

Pr0gram4tez: And, as you've stated, the keys are in the relational store. All of them?

Nubimus123: Quite true.

Pr0gram4tez: So in the event of a compromise, beg the gods not, an adversary, perhaps an agent of Syracuse, would have all the tools necessary to recover the plaintext and use it against you or your clients or Athens itself?

Nubimus123: Well, I suppose so, Programetes. That seems unlikely, though. The attacker would have to compromise multiple, higher-level layers to achieve access.

Pr0gram4tez: What evidence do you have that any other layer of your application isn't equally vulnerable? Have you not used the same strategy of trusting the framework above the judgement of mortal programmers? Is not the desired result of cryptography to protect your clients data in the event the higher levels of the application are compromised?

Nubimus123: Well, I fear that may be true, rare friend.

Pr0gram4tez: Tell me, then. Is the cryptography not potentially rendered irrelevant by your key storage practices. Is the net result that there is little security benefit but there is complexity incurred in the application?

Nubimus123: I admit that my understanding, and perhaps that of the whole team, of how to implement a cryptographic system was inadequate. We became emboldened by the code crafted by the gods and it made us feel invincible. I now know that a tool of the gods in the hands of man doesn't make that same man godlike.

Pr0gram4tez: I regret that our conversation may leave you with your mood diminished, Nubimus. I am sorry for that. It was not my intention to play your adversary, but I am thankful that you and your clients may benefit.

Nubimus123: And I thank you for it, teacher Programetes. Regardless, my eyes can focus on pixels no longer. It's time to close this chat window.

Pr0gram4tez: Beer, then?

Nubimus123: I've changed my mind, Programetes. I feel I must stay sharp for a busy weekend of study.

Sat Mar 10 2012 11:50:32 GMT+0000 (UTC)

The Sylvester Matrix Library Comes to Node.js

Before some recent machine learning implementations in node I set out to find a reasonable matrix math/linear algebra library for node.js. The pickings were slim but I managed to dig up a general JavaScript matrix math library written by James Coglan called sylvester. Clearly, sylvester had to be node-ified.

With the help of Rob Ellis (a collaborator on natural) it's been wrapped up into a node project titled node-sylvester & NPM and has had some features added such as element-wise multiplication, QR, LU, SVD decompositions and basic solving of systems of linear equations.

In this post I'll cover some of the basic structures and operations supported by sylvester, but it will by no means be complete. I'll focus solely on the Matrix and Vector prototypes, but sylvester also supports Line, Plane, and Polygon. Also within the covered prototypes I'll only demonstrate a small, but useful subset of their functionality.

Currently, the only reasonable sources of documentation for functionality existing only in the node port are in the README while general sylvester functionality is covered in its API docs. In time I will (hopefully with the help of the community) provide some more complete documentation.

Installation

Getting ready to use the node port of sylvester is what you'd expect, a standard NPM install.

npm install sylvester

You can then require-up sylvester in your node code and use the prototypes within.

var sylvester = require('sylvester'),
  Matrix = sylvester.Matrix,
  Vector = sylvester.Vector;

Vector and Matrix Prototypes

Matrices and vectors are abstracted by the Matrix and Vector prototypes.

Instances can be created using their create functions and passing in an array of values. Vector.create accepts a one dimensional array of numbers and Matrix.create accepts multiple dimensions.

var x = Vector.create([1, 2, 3]);
var a = Matrix.create([[1, 2], [3, 4]]);

representing:

Global shortcuts exist to clone Vector and Matrix prototypes with $V and $M respectively. The following is the semantic equivalent of the previous example.

var x = $V([1, 2, 3]);
var a = $M([[1, 2], [3, 4]]);

Matrix's Ones function will create a matrix of specified dimensions with all elements set to 1.

var ones = Matrix.Ones(3, 2);

Matrix's Zeros function does the same except with all elements set to 0.

var zeros = Matrix.Zeros(3, 2);

An identity matrix of a given size can be created with Matrix's I function.

var eye = Matrix.I(4);

Data Access

Values can be retrieved from within a Matrix or Vector using the e method.

a.e(2, 1); // returns 3
x.e(3); // returns 3

The entire set of values can be retrieved/manipulated with the elements member of Matrix and Vector. elements is an array-of-arrays-of-numbers in the case of matrices and it is a simple array-of-numbers for vectors.

var a_elements = a.elements; // [[1, 2], [3, 4]]
var x_elements = x.elements; // [1, 2, 3]

Basic Math

Many standard mathematic operations are supported in Vector and Matrix clones. In general the arguments can either be scalar values or properly-dimensioned Matrix/Vector clones. While not covered here element-wise versions of many multiplicative/specialized operations are also available.

Matrices and vectors can be added-to and subtracted-from by scalar values using the add and subtract functions.

var b = a.add(1);
var d = a.subtract(1);

symbolizing:

Naturally, like-dimensioned matrices and vectors can be added to and subtracted from each other.

var e = a.add($M([[2, 3], [4, 5]]));
var f = a.subtract($M([[0, 2], [3, 3]]));

meaning:

Matrices/vectors can be multiplied with matrices/vectors of appropriate dimensions.

var g = a.x($V([3, 4]));

The dot-product of two vectors can be computed with Vector's dot method.

var y = x.dot($V([4, 5, 6]));

depicting:

Transposing

A Matrix can be transposed with the transpose method.

var at = a.transpose();

where at represents the matrix:

Segmentation/Augmentation

The first n elements can be removed from a Vector with the chomp method.

var n = 2;
var xa = x.chomp(n);

In contrast the first n elements can be retrieved with the top method.

var xb = x.top(n);

Vectors can have a list of values appended to the end with the augment method.

var xc = x.augment([4, 5]);

Matrices can have a column appended to the right with Matrix's augment method.

var m = a.augment($V([3, 5]));

A sub-block of a Matrix clone can be retrieved with the slice method. slice accepts a starting row, ending row, starting column and ending column as parameters.

var aa = $M([[1, 2, 3], [4, 5, 6], [7, 8, 9]]);
var ab = aa.slice(1, 2, 1, 2);

The code above produces a matrix ab shaped like:

Decompositions/Advanced functions

A small number of decompositions are currently supported. These have all been recently added and at the time of writing are, while functional, not necessarily as computationally efficient as they could be. Also their stability cannot be guaranteed at this time.

Note that these were all implemented in pure JavaScript. My personal hope is to keep node-sylvester 100% functional in pure JavaScript to maintain the best compatibility across all platforms. In the long run, however, I hope to employ time-tested, computationally-efficient native libraries to perform these operations if the host machine allows.

A QR decomposition is made possible via Matrix's qr method. An object is returned containing both the Q and the R matrix.

var qr = a.qr();

resulting in the object:

{ Q: 
   [-0.316227766016838, -0.9486832980505138]
   [-0.9486832980505138, 0.3162277660168381],
  R: 
   [-3.162277660168379, -4.427188724235731]
   [5.551115123125783e-16, -0.6324555320336751] }

Singular Value Decompositions (SVD) can be produced via Matrix's aptly named svd method. An object is returned containing the U (left singular), S (singular diagonal), and V (right singular) matrices.

var svd = a.svd();

which returns:

{ U: 
   [0.40455358483359943, 0.9145142956773742]
   [0.9145142956773744, -0.40455358483359943],
  S: 
   [5.4649857042190435, 0]
   [0, 0.36596619062625785],
  V: 
   [0.5760484367663301, -0.8174155604703566]
   [0.8174155604703568, 0.5760484367663302] }

Principal Component Analysis can be performed (dimensionality reduced) and reversed (dimensionality restored with approximated values) using Matrix's pcaProject and pcaRecover methods respectively.

pcaProject accepts the number of target dimensions as a parameter and returns an object with the reduced data Z and the covariant eigenvectors U.

var pca = a.pcaProject(1);

{ Z: 
   [2.2108795577070475]
   [4.997807552180416],
  U: 
   [0.5760484367663208, -0.8174155604703633]
   [0.8174155604703633, 0.5760484367663208] }

and pcaRecover approximately recovers the data to the original dimensionality.

pca.Z.pcaRecover(pca.U);   

which produces:

Solving Linear Equations

Simple linear equations in the form of

can be solved using the suitability-named solve method.

var A = $M([
    [2, 4],
    [2, 3],
]);

var b = $V([2, 2]);
var x= M.solve(b);

which is representative of

Conclusion

Thanks to James Coglan's hard work on sylvester plenty of matrix math functionality has been exposed to JavaScript. With a little extra work we've been able to expose quite a bit more to node. There's still more work to do not only in optimization but also in capabilities. In time hopefully node-sylvester will become fast and complete.

As a reminder, the functionality outlined above is not representative of the full list of current capabilities. Between the README in the source and the sylvester API documentation you should be able to get a reasonably clear picture. Still, if you're looking for ways to help with the project a concerted documentation effort would be wonderful and appreciated.

Sat Dec 17 2011 05:00:00 GMT+0000 (UTC)