TestIDs
---------
1
2
3

TestData
--------
A
B

What I want is a join that produces:

Result
--------
1 A
1 B
2 A
2 B
3 A
3 B

This is essentially a join without an "on" clause, however the trick in SQL is that the "join" keyword isn't needed either.  The select statement is simply:

select * from TestIDs, TestData

Notice that there is a comma seperating the tables to select from, and no join keyword anywhere.  Handy!

Some data structures (DTO's to some folk, and classes as well) have a large number of properties with the same return type, such as:

public struct MyData
{
public MyData(string firstName, string middleName, string lastName, string surName, string formerName)
{ // set the properties }
public string FirstName { get; set; }
public string MiddleName { get; set; }
public string LastName { get; set; }
public string SurName { get; set; }
public string FormerName { get; set; }
// ...and so on...
}

Imagine that this strucuture is implemented with field-backed properties (instead of automatically-implemented properties) and we wanted unit tests to ensure that setting SurName didn't change the backing field for FormerName.  Each unit test would need to instantiate an instance of MyData by passing a laundry-list of parameters.  Here's a sample of a unit test (and accompanying BuiltTarget() method) showcasing how ugly this can be (notice that BuildTarget() is an ungainly mess; and that there would normally be one or more tests for each parameter).

Public Function BuildTarget(ByVal firstName As String, _
ByVal middleName As String, _
ByVal lastName As String, _
ByVal surName As String, _
ByVal formerName As String) As MyData
Return New MyData(firstName, middleName, lastName, surName, formerName)
End Function

<TestMethod()> _
Public Sub New_WhenFirstNameProvided_KeepsValue()
Dim firstName = "Steve"
Dim target = BuildTarget(firstName, _
Nothing, _
Nothing, _
Nothing, _
Nothing)

Dim expected = firstName
Dim actual = target.FirstName

Assert.AreEqual(expected, actual)

End Sub

To work around this my tests contain a single "BuildTarget()" factory which declares each parameter as optional, and provides a default value.  Then when I need to set just a single parameter I can do so by name and let the default values provide the remaing parameters.  The above sample cleans up to look like:

Public Function BuildTarget(Optional ByVal firstName As String = Nothing, _
Optional ByVal middleName As String = Nothing, _
Optional ByVal lastName As String = Nothing, _
Optional ByVal surName As String = Nothing, _
Optional ByVal formerName As String = Nothing) As MyData
Return New MyData(firstName, middleName, lastName, surName, formerName)
End Function

<TestMethod()> _
Public Sub New_WhenFirstNameProvided_KeepsValue()
Dim firstName = "Steve"
Dim target = BuildTarget(firstName:=firstName)

Dim expected = firstName
Dim actual = target.FirstName

Assert.AreEqual(expected, actual)

End Sub

As you can imaging, the small amount of extra code in the BuildTarget() method is insignificant compareed to the savings in each TestMethod.

At work we are writing a .NET 2.0 application that has its roots back in 1.1.  Some ancillary applications were also written in 1.1, and shared some common libraries; so when it came time for this application to be upgraded to 2.0 some libraries had to be left behind for compatibility reasons.

One such library has the task of loading data from databases, and was designed to read from MS Access, SQL Server, or MSDE; however in the ongoing development of our application we are scraping the MS Access use in favor of SQL Compact Edition.  For those familiar with SQL CE, you'll know it's not compatible with the 1.1 framework--so we were faced with the issue of either duplicating the functionality of the 1.1 DLL in a 2.0 DLL, so that old applications could be left alone, or with doing massive work to overhaul the old applications and try to bring them all up to 2.0...  neither option was terribly appealing (although the first option was obviously preferable).

Our project architect put the task to me, and had a plan of attack that he figured would let us make the 1.1 DLL talk to the SQL CE database via a 2.0 DLL proxy--however you'll know that 1.1 projects cannot reference 2.0 projects.  Here's where a little trickery comes in...

For a quick recap we have: A Framework 2.0 application, which calls a 1.1 Library to retrieve data.  We need: A Framework 2.0 application to call the 1.1 Library to call a 2.0 Library to retrieve the data from SQL CE.

To achieve this we added an interface to the 1.1 Library, and implemented that interface in the new 2.0 library which uses the interface to fetch the needed data from the database.  The 1.1 library uses reflection to late-load the 2.0 library and instantiates it by its interface.

So the workflow is:

• 2.0 Application makes a call to the 1.1 Library
  •  1.1 Library uses an Assembly.Load() call to late-load the new 2.0 Library and casts it to the 1.1's interface definition.
  • 1.1 Library then can call the Interface member, which...
    • Talk through the 2.0 framework to our SQL CE database.

The suspect was immediately either the IDE controller or both of its drive. After transplanting the drives to a seperate machine and failing to find their partitions, and after trying a new drive in the machine it was looking as though possibly all were fried.

A couple months have passed and my landlord has had me build him a new computer and the day has come to try to recover his data... Below is the process I've used to diagnose and recover his files.

1) Attempt to image the entirety of the drive

With the physical stability of the drives in question, the first order of business was to mitigate the amount of work done against the failed drives themselves. Windows claimed the drives were unformatted, and linux would not let me mount them (although their partition tables were visible).

The weapon of choice was dd, a data-copy utility that works at a low level and with a little bit of knowledge can be an invaluable tool. dd can be extremely dangerous if you aren't careful with it!!! dd is basically a data-copy tool which takes a couple arguments; "if" (input file) and "of" (output file). The device/file you set "if" to will be copied to the "of" device/file. You can add more parameters to control the amount of data copied, and the blocksize that the harddrive's partition uses.

Example: Backing up a drive's partition table

The drive partition table is stored in the first kilobyte (not certain this is always true, it may depend on the block size) of the partition. To make a backup:

dd if=/dev/[drive] of=$HOME/[drive]_part.bak bs=1k count=1

This will copy the first kilobyte (bs=1k count=1) from /dev/[drive] to a file called [drive]_part.img in the executing user's home directory.

Back to data recovery with dd... to make an image of the entire drive I used the following command, note that this creates a file equal to the physical size of the disk, so be sure you have enough room. i.e. if you have a 1GB partition on a 100GB drive (and 99GB unpartitioned) the resulting image will be 100GB.

dd if=/dev/[drive] of=$HOME/backups/[drive].dmg bs=512 (NOTE: bs varies depending on the filesystem settings, common sizes are 512, 1024, and 4096, although many others are possible. One way to determine the block size is to try copying one 512b block, one 1k block, and one 4k block (so on so forth). Wrong block sizes should result in an error, ruling out the selection.

dd does not work through errors by default, so if it finds bad sectors or other obstacles it will error out--providing evidence of the type of failure on your hands. I lucked out, and dd ran through the full 20GB drive without error which told me that the drive failure was likely with the partition or filesystem and not a hardware failure. If dd fails due to drive errors such as bad sectors you can use ddrescue or gddrescue, both of which use dd and will replace errors with null (nothing) values in the image, resulting in a backup image of as much data as possible. One other very cool feature is if you use these tools to copy from a device that is inconsistently reading it's contents then you can run the tool multiple times, and if a 2nd (or 3rd, 4th...) pass is able to successfully read what a previous pass failed at it will fill in the void left by the earlier passes due to the error. This proves wonderfully useful on scratched CDs, among other things.

Now for the real fun: With the disk image in place I shutdown and remove the physical drive from the machine (so that if it is a hardware issue, any further damage will be prevented. My landlord was considering having the drive professionally recovered if we were unsuccessful, so we wanted to make sure we did no further harm to it). Upon booting back up I set the drive image to a loopback device, which lets you mount a disk image like it were a regular drive (iso's work great with this)

2) Attempt to mount the image

After unplugging and removing the dead drive I tried to mount the image I'd just made:

mount /home/steve/[drive].dmg -o loop -t vfat /mnt/recovery

But it failed, barking about a bad superblock. This was starting to look more like a bad partition table than a fried drive--so back in went the drive.

3) A great utility

After some Googling I found multiple references to a utility called testdisk which is a free drive recovery tool. My only beef was that it only seems to work on physical drives, not loopback images (like what I made above). So with the drive back in I ran testdisk, and after selecting the appropriate drive and type from it's menus it immediately told me I had a bad partition table.

Using the restore and scan options of testdisk brought the drive totally back to life! I was very impressed by it, as it seems to have a lot of power under the hood but not a lot of difficulty in it's user-interface.

The drive lived! I saved it's updated partition tables back to the drive, rebooted as it recommened and was able to successfully mount the partition.

4) Get everything off the drive

There was no sense running the risk of letting the drive cause more problems, especially as there was over a Terabyte of free space on the new machine. So I rsync'd everything off of it immediately:

rsync -av /mnt/[mountpoint] /var/storage/recovered/

5) Shred the drive

The old 20GB drive was recovered, and my landlord wanted to recycle it (give it away). So first we wanted to shred any remnants of it's contents. There is a GNU utility aptly named "shred" that does specifically this.

shred /dev/[drive]

And many hours later (shred defaults to 25 passes, which is extreme but can be changed) the drive was wiped clean.

The UUIDs of my new installation were lost, so after a little Googling I found that there's a helpful little utility called "vol_id" specifically for determining this type of info...

vol_id [partition]

and voila, I've got my fstab back in fine shape

Objective: Gain entry to your dwelling without having to a) wake up the neighbor or b) explain to a cop that your ID is just behind the door you're trying to jimmy.

*My* Solution (your mileage may vary): Perform a preliminary port-scan before proceeding to more intensive penetration testing. Start with all the doors... my main door is around back, and it leads to the three apartments in the house I live in. It happens to be unlatched (rare, as it has a hydraulic ram to close it and automatically locks) however the inner door to my apartment is locked tight. I've no credit card handy to attempt to pop the lock, however it would be futile--I've tried before.

On to the front door; it's a shaky old thing that lacks a locking knob, has two cheap-o mini-deadbolts and a sliding chain lock to hold it tight. At first try it's shut and locked, no luck here. The two windows to the apartment are also unhelpful, one is locked and the other has an AC unit bolted in.

Back to the front door. I shouldered it pretty firmly and was surprised when it gave! Turns out the wood has swollen with the humidity to where only 1 of the 2 little deadbolts catch, and only by a small amount. So I'm left with a sliding chain lock between me and bed. The chain lock is installed correctly though, and the door must be closed for it to be locked or unlocked. fuck

Final solution: Scavenge, fortunately there's a construction site in the back yard (they're building a new garage) and I manage to find some twine. I cut off about 6' (with a utility knife I found) and tie one end around the end of the chain, on the little knob piece that slides in the rail and actually locks the door. Then I run the twine up the inside of the door (inside the apartment, same side as the lock), over the top, and down the front. Pull the door to where it almost latches and start tugging on the twine. Amazingly it only took about 10 seconds of fiddling to pop the chain out of it's rail, and voila!

Maybe later I'll post about a string-&-baseball injection attack on a large leaf-server (ie: Tree) which holds (another) toy helicopter problem.

Since the source code I had on my laptop was within a Virtual Machine, I thought it would be a good solution to make an encrypted filesystem big enough for the VM, and only mount it when I wanted to work.

Here's my requirements:
- Encrypted FS that is (un)mountable whenever need be
- Passphrase to mount the filesystem
- 30GB of storage within the filesystem, to accomodate the 30GB VM disk.

First, we create a file that will become the filesystem. Using dd we make it the size we want:
dd if=/dev/zero bs=1G count=30 of=/home/steve/devel_image

Here's what the attributes mean:
if is the input file since we're making an empty file we'll set if to /dev/zero, which will give us an empty file of zeros
of is the output file, or the file you want to create. Here I'm telling it to make a file called devel_image in my /home/steve folder
bs is the bytes, but I like to call it the bucket size, since the way it is used is to declare your unit size. Here 1G means we wants 1GB chunks, or buckets. You can give it different units, 1M, 5G, whatever.
count is the number of buckets you wish to use. Since we want a 30GB file, and are using 1G buckets, we'll need 30 of them to make 30GB.

This makes a 30GB file called devel_image in my home directory, and takes some time to complete (30 gigs is kinda big, y'know?)

Next we setup a loopback device (which will make the system see the file as a drive). using losetup you can create the device, and you can even add encryption at this point--however it's becomming deprecated, and better to handle the encryption on it's own (as cryptoloop support will be removed from the kernel at some point

Shred can be used to wipe files, or whole drives. To wipe a drive, the following parameters are good to know
-z After shredding, one more pass will be made to write all zeros to the drive, not necessary, but it hides the fact that the disk has been shredded
-v Display messages about progress, which is handy since shredding a drive takes a fair deal of time
-n x x is the number of passes you wish to make. 3 is a good number, and if you use the -z option it will be 3 shredding passes followed by a 4th pass to zero out all the data

Put it together and pass it a drive to wipe:
shred -zvn 3 /dev/sda

And voila, all your private data goes bye-bye

Doing a 'cat /proc/mdstat' produced:
Personalities : [raid6] [raid5] [raid4]
<none>

Thankfully I had backups of my mdadm.conf and lvm.conf files so I knew that as long as I didn't alter the data on the drives I could always fall back to Edgy and still have my files.

After trying a few things with mdadm, all unsuccessfully, I decided to uninstall it entirely. So I made a (second) backup of my mdadm.conf and lvm.conf files, and did a:
apt-get --purge remove mdadm lvm2

This uninstalls the packages and also removes the config files (that's what the --purge option is there for).

After uninstalling them I verified the configs were gone and reinstalled the packages through Synaptic inside of GNOME. During the install it prompted which arrays were required to boot. On my failed upgrade I had configured it with "all" but I really didn't need my array at boot time, my root partition is on a standalone IDE drive, so this time I told it "none".

I expected to have more tinkering to do, but after the install I again did a "cat /proc/mdstat" and to my surprise my array was back! *yay*

Now I just had to get my lvm partitions back. I've found the Gentoo LVM2 installation guide very usefull, and a good resource for LVM, so I turned to it to see how to recover volumes. The commands to restore my volumes were:
vgscan
vgchange -a y

And voila, my array and it's partitions were good as new. A quick "mount -a" brought them all up without a glitch!

The process is easy-as-pie. Just run "update-manager -c -d" and it's a simple walk-through process.

Obligatory Warning: Feisty won't be officially released until April, and it's still labeled an Alpha release, so be warned it has lots of bugs and is not stable--personally I see my gnome-panel crash quite often, as well as a number of other oddities. You've been warned