Latest Entries »

Here’s a collection of find usages, if you have any to add post a comment and I’ll add them..

Enjoy

One of the way’s I’ve used the find command to move or copy large quantity of files in sub directories to a new destination directory.

    find . -type f -exec cp {} /new/directory \;

You can also use find to count the number of files in a directory recursivly.

    find . -type f | wc -l

You can also count the number of sub directorys by changing the type to d.

    find . -type d | wc -l

This is how to mount a local ISCSI extension image as if it were an actual physical drive.

First make sure you disable access to this ISCSI target as you could irrevocably damage your data trying to access it twice.

Then create a virtual pointer using mdconfig

    mdconfig -a -f /mnt/Storage/extents/image.img

The above command will return a device name eg md3

Now Load the fuse module

    kldload /usr/local/modules/fuse.ko

Create a mount point for the drive and mount it using ntfs-3g

    mkdir /mnt/mountpoint
    ntfs-3g /dev/md? /mnt/mountpoint

Unmount the drive.

    umount /mnt/Storage/mountpoint

Unlink the device from the file.

    mdconfig -d -u md?

Check out the FreeNAS Documentation
Check out mdconfig man page.

If you have any questions or tips please post them, I’ll try and get back to you.

Tmux interact with any or all panes

Situations I find this useful are that you sometimes need to interact with multiple servers by sending them the same command, I hear you say that tmux can do that with synchronize-panes you would be partially right it does send the same input to all panes which can be useful, and other times this can be annoying when you want to have multiple panes open viewing output from multiple commands and also want the ability to send individual panes a command and or multiple panes the same command.

This is a pair of scripts I use to capture the input of one pane running a bash script and send the input a ‘command’ to a configurable amount of other panes.
With a little modification this could be used to send commands to multiple panes across windows, sessions and possibly even servers.

This might not work as expected or at all with your tmux session setup, if you do encounter problems try posting your questions and I’ll try and get back to you.

In the mean time check out the tmux man page here.

Enjoy.

#!/bin/bash
###################################################################
# author Damien Roche
# created 10 October 2011
#
# Create a 3 pane layout two side by side over one horizontal
#
#	Layout
#	_________________
#	|       |       |
#	|_______|_______|
#	|_______________|
#
###################################################################
SESSION=$USER
WINDOW='copycat'

cd ~/
clear

tmux new-window -t $SESSION:9 -k -n $WINDOW
tmux split-window -t "$SESSION:$WINDOW.0" -v
tmux split-window -t "$SESSION:$WINDOW.0" -h

tmux send-keys -t "$SESSION:$WINDOW.1" "bash ~/bin/capture.sh $SESSION $WINDOW 0 2" C-m

#!/bin/bash
#################################################################
# author Damien Roche
# created 10 October 2011
# 
# Call this script after you initialize your window & panes
#
# Captures input 'command' from the current pane and send it to an 
# array of panes received from the window initialize script
#
# If you need to interact with more than 6 panes at a time
# you'll need to capture them from the arguments 
#
#################################################################
clear
if [ $# > 3 ]; then
	SESSION=$1
	WINDOW=$2
	PANES=( $3 $4 $5 $6 $7 $8 )
	while : 
	do
		read command
		for PANE in "${PANES[@]}"
		do
			tmux send-keys -t "$SESSION:$WINDOW.$PANE" "$command" C-m
		done
	done
fi

FreeNAS User & Group Management

Add User

Adding a new user in FreeNAS is an easy task, simply open the Add User pane by selecting the Add User option on the left hand menu.

Now enter all the details for the user you wish to add

Username
The users name genraly. This field is case sensitve.
Primary Group
The group this user belongs to. Default a group will be created for each user. This field has advanced use cases explained later. Mostly will be left as is default.
Home Directory
Defaults to a non existent directory, If you wish your users to have a directory you can change this here. Eg: /mnt/Storage/home/$USER
Shell
The shell assigned to the user upon SSH login.
Full Name, Email, Password and Password confirmation
Are self explanitory fields
Disable logins
If you need to suspend an account later you can check this option.

Note
Fields you genrally don’t need to modify or worry about User ID, Home Directory, Shell

Users & Groups Example

Take the following as an example. You have a group of users John & Jane Doe and you want them both sharing a single dataset. We could setup one group ‘Documents’ and make them both members and they will inherit access to the ‘Documents’ dataset as member of the ‘Documents’ group.

Implementation

Were going to add a couple of initial user accounts. Where $USER1 & $USER2 are your aditional accounts.

Users

Administrator Generic Owner user

  • Guest For guest access
  • $USER1 janedoe
  • $USER2 johndoe

Groups

  • Documents Group Only

Next we add the dataset where we wish to store the files for the Documents User. We also change the permissions on this dataset to allow the group to write to it.

We now create a CIFS share point called Documents.
This is where you will access the share eg: //netbiosname/Documents
You can also substitute the IP address of the server eg: //192.168.1.1/Documents

Now we add the $USER1, $USER2 to be members of the Documents Group, So that they inherit the permissions and are authorized to read and write to the shared folder.

FreeNAS is a FreeBSD based Network Attached Storage Operating System, I like to think easy to use fast and flexible and very powerful.

Big thanks to the FreeNAS team for providing us with an exceptional piece of software and lots of cool features to play with ( and break ). Also thanks to the users on freenode#freenas for putting up with my constant questions and poking.

Setup Replication

The naming scheme I’m using is master-host [ The host to replicate from ] and slave-host [ The host to replicate to ]

Enable ssh + root password logins
This is a potential security risk ( we can disable the password logins after we are done. )

Go to the Services pane
Toggle SSH to on and edit the settings to reflect

FreeNAS SSH

FreeNAS SSH pane

In your terminal connect to the master-host and copy the replcation public cert from the active replication server to the backup replication server.


ssh root@[master-host-ip]
scp /data/ssh/replication.pub root@:/tmp/

Now on the master-host & slave-host create the ~/.ssh directories


mkdir ~/.ssh

On the slave-host copy the contents of the replication.pub cert we copied over to the authorized_keys file which will be located in your ~/.ssh/ directory and change the permissions on the authorized_keys file.
( this insures we get the key import in one line, and we restrict access )


mount -uw /
cat /tmp/replication.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Now on the master-host we are going to connect to the slave to check it’s working and also confirm each server to the known_hosts file. For this to persist reboots we need to mount the file system rw.


mount -uw /
ssh -i /data/ssh/replication

FreeNAS SSH Known Host confirmation

You will be asked to accept the host your are connecting to ( choose yes ).

Note: If you have a multi-homed setup you’ll need to repeat the above command for all addresses you want to do replication over.

GUI Configuration
To setup replication we will need to create a periodic snapshot task to replicate to the slave.

From the FreeNAS GUI enter the Storage pane and select “Periodic Snapshot Tasks” Shown below.

FreeNAS Periodic Shapshot creation

Create a new task with parameters relevant to your needs, below is an example of mine

FreeNAS Periodic Snapshot Task properties

Now to create the ZFS Replication, On the Storage pane select “ZFS Replication” shown below

FreeNAS ZFS Storage Replication

Storage Replication Pane

Add Replication

FreeNAS Add Replication

Add new Replication

  • Mount Point: The local dataset that has a replication task set you want to replicate
  • Remote ZFS filesystem: The remote volume name you want to replicate to
  • Remote hostname: IP or Hostname of the replication
  • Remote hostkey: The public key of the remote host to replicate to ( see below )

How to get the remote host public key


ssh root@[remote-host]
more /data/ssh/replication.pub

copy this output to your favorite text editor

Note: It should be all one one line and NOT split
When your sure it’s not on multiple lines copy it to the Remote hostkey field on the Add Replication Window.

Now review your settings and hit Ok.

We were recently encountering elevated errors with haproxy returning a 502 Bad Gateway response to external http requests.

Upon further investigation it seemed as if the proxy was converting an upstream 302 response into 502 errors at the Gateway.

After checking the logs and headers, I noticed that the header uri was relative and should be absolute for http 1.1 eg: http://sub.example.com/page.ext so when the 302 header uri was set to absolute it resolved the problem.

I’m not too sure if this is the primary cause as this error was only happening for the occasional visitor, and I could not reproduce the error. We only had one user effected by this issue.

Other info that might help
http://www.formilux.org/archives/haproxy/0810/1447.html

I finally got around to fixing my dual head setup on Ubuntu 11.04 Natty

The problem I was having was one monitor was not being detected and when I tried a manual setup it would not accept a resolution greater than 640×480 px which was really annoying as you can imagine.

Yes I’m using the proprietary nvidia drivers which taints the linux kernel … but nouveau isn’t quite up to scratch still experimental at the moment. I will revise this when and if I do switch.

Edit xorg.conf

    sudo gedit /etc/X11/xorg.conf 

I added the following lines to xorg.conf. I didn’t have much time to tweak the setup once it worked as this is my main development system and I needed it for work. I’m sure there are unneeded options here and I will get around to testing more extensively in the near future I hope.

Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Screen0"
EndSection

Section "Files"
EndSection

Section "Monitor"
    Identifier     "Monitor0"
    VendorName     "Dell"
    ModelName      "U2311H"
    HorizSync       28.0 - 33.0
    VertRefresh     43.0 - 72.0
    Option         "DPMS"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "Quadro NVS 295"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
    Option         "TwinView" "True"
    Option         "ProbeAllGpus" "True"
    Option         "DynamicTwinView" "True"
    Option 	   "ConnectedMonitor" "DFP-1, DFP-0"
    Option         "MetaModes" "nvidia-auto-select, nvidia-auto-select"
    Option         "SLI" "Off"
    Option         "CustomEDID" "DFP-1:/home/daxroc/edid/dell.bin"
    SubSection     "Display"
        Virtual     3840 1080
        Depth       24
    EndSubSection
EndSection

Update
I was also getting intermittent screen flicker which ended in one monitor being detected. All symptoms were resolved when using display port to display port I can even remove the xorg.confirm

Leave a comment if you find any other problems with this setup, Hope it helps
Big thanks to @mordy on freenode, appreciate the help.

Only one instance

I came across a pair of very useful scripts today,

solo.pl by Tim Kay -> http://timkay.com/solo
In short solo only allows one instance of a script to run. Which can be extremely useful

and watcher.py https://github.com/greggoryhz/Watcher

I am currently using solo together with watcher.py for a simple http media file server mirroring

How?

Cron

# Solo cron watcher 
*    *    *    *    *    ~/bin/solo.pl -port=<port> ~/bin/Watcher/watcher.py start

*Note: Use a port greater than 1023 if your not using root.

Watcher recursively monitors a directory for changes and executes a command when changes have been made.

So with solo + watcher + rsync + ssh we have a simple secure near real-time file mirroring.

If you wanted for example to cache media.example.com and redirect to an alternate back-end from your www.example.com

You could use something like

frontend main-application
    mode http
    bind <ip>:80
    option httplog
    acl is_media hdr_dom(host) media
    use_backend media if is_media
    default_backend applcation

backend application
    mode http
    option httplog
    option httpclose
    option forwardfor
    option httpchk HEAD /check.txt HTTP/1.0
    stats enable
    stats auth someuser:somepassword
    balance roundrobin
    server a-web 192.168.1.50:85 cookie a check
    server c-web 192.168.134.51:85 cookie c check
    server d-web 192.168.134.52:85 cookie d check

backend media
    mode http
    option httplog
    option forwardfor
    option httpchk HEAD /check.txt HTTP/1.0
    server media-b 192.168.1.100:85 cookie a check
    server media-a 192.168.1.101:85 cookie a check</p>

And if you wanted to redirect from a legacy port to another sub domain you could use


frontend main-application
	mode http
	option httplog
	bind <ip>:80
	bind <ip>:2000
	option httpclose
        option forwardfor
	acl is_legacy 	dst_port 2000
	use_backend legacy if is_legacy	
	default_backend application


backend application
	mode http
        stats enable
        stats auth someuser:somepassword
        option httpclose
        option forwardfor
	option httpchk HEAD /check.txt HTTP/1.0
	balance roundrobin
	server a-web <ip>:80 cookie a check
	server b-web <ip>:80 cookie a check
	server c-web <ip>:80 cookie a check


backend legacy
	mode http
	option httpclose
	option forwardfor
	option httpchk HEAD /check.txt HTTP/1.0
	server redirect <ip>:80 redir http://example.example.com	check

GW,Galway,Connacht
LM,Leitrim,Connacht
MO,Mayo,Connacht
RN,Roscommon,Connacht
SO,Sligo,Connacht
AM,Antrim,Ulster
AH,Armagh,Ulster
CN,Cavan,Ulster
DL,Donegal,Ulster
DN,Down,Ulster
DY,Derry,Ulster
FH,Fermanagh,Ulster
MN,Monaghan,Ulster
TE,Tyrone,Ulster
KY,Kerry,Munster
C,Cork,Munster
CE,Clare,Munster
LK,Limerick,Munster
TN,Tipperary N,Munster
TS,Tipperary S,Munster
WD,Waterford,Munster
D,Dublin,Leinster
CW,Carlow,Leinster
KE,Kildare,Leinster
KK,Kilkenny,Leinster
LS,Laois,Leinster
MH,Meath,Leinster
OY,Offaly,Leinster
LD,Longford,Leinster
LH,Louth,Leinster
WH,Westmeath,Leinster
WW,Wicklow,Leinster
WX,Wexford,Leinster

Follow

Get every new post delivered to your Inbox.

Join 40 other followers