Linux : What does ‘source’ command do

$ source activate

What does ‘source’ do ?

Source is a bash shell built-in command that executes the content of the file passed as argument, in the current shell

Running the command source on a script executes the script within the context of the current process. This means that environment variables set by the script remain available after it’s finished running. This is in contrast to running a script normally, in which case environment variables set within the newly-spawned process will be lost once the script exits.

You can source any runnable shell script. The end effect will be the same as if you had typed the commands in the script into your terminal.

Linux : SFTP command to transfer transfer files

Hi Friends,

SFTP stands for secure File Transfer Protocol, which help to transfer files over network in a secure / encrypted fashion. It is mainly used when you have to upload or download a file to a remote machine on network. It is similar to SCP, however it gives you more options to navigate ie cd , mkdir , changing file permission, ls etc while working with file.

$ sftp 1.244.344.444
sftp >

Now you are connected to remote server and do ls and list remote file (can use other commands as well)
Append “l” in front of command to list local files
eg. $ lls => local list

$lpwd => local present working directory
$lmkdir => make directory in local box

$ get filename => will download from remote to local directory
$ put filename => will upload from local to remote directory

Detailed Example :

1. How to Connect to SFTP

By default, same SSH protocol is used to authenticate and establish a SFTP connection. To start an SFTP session, enter the username and remote hostname or IP address at the command prompt. Once authentication successful, you will see a shell with an sftp> prompt.

[root@tecmint ~]# sftp tecmint@27.48.137.6

Connecting to 27.48.137.6...
tecmint@27.48.137.6's password:
sftp>

2. Getting Help

Once, you in the sftp prompt, check the available commands by typing ‘?‘ or ‘help‘ at command prompt.

sftp> ?
Available commands:
cd path                       Change remote directory to 'path'
lcd path                      Change local directory to 'path'
chgrp grp path                Change group of file 'path' to 'grp'
chmod mode path               Change permissions of file 'path' to 'mode'
chown own path                Change owner of file 'path' to 'own'
help                          Display this help text
get remote-path [local-path]  Download file
lls [ls-options [path]]       Display local directory listing
ln oldpath newpath            Symlink remote file
lmkdir path                   Create local directory
lpwd                          Print local working directory
ls [path]                     Display remote directory listing
lumask umask                  Set local umask to 'umask'
mkdir path                    Create remote directory
put local-path [remote-path]  Upload file
pwd                           Display remote working directory
exit                          Quit sftp
quit                          Quit sftp
rename oldpath newpath        Rename remote file
rmdir path                    Remove remote directory
rm path                       Delete remote file
symlink oldpath newpath       Symlink remote file
version                       Show SFTP version
!command                      Execute 'command' in local shell
!                             Escape to local shell
?                             Synonym for help

3. Check Present Working Directory

The command ‘lpwd‘ is used to check the Local present working directory, whereas ‘pwd‘ command is used to check Remote working directory.

sftp> lpwd
Local working directory: /
sftp> pwd
Remote working directory: /tecmint/
  1. lpwd – print the current directory on your system
  2. pwd – print the current directory on the ftp server

4. Listing Files

Listing files and directories in local as well as remote system.

On Remote
sftp> ls
On Local
sftp> lls

5. Upload File

Put single or multiple files in remote system.

sftp> put local.profile
Uploading local.profile to /tecmint/local.profile

6. Upload Mutiple Files

Putting multiple files on in remote system.

sftp> mput *.xls

6. Download Files

Getting single or multiple files in local system.

sftp> get SettlementReport_1-10th.xls
Fetching /tecmint/SettlementReport_1-10th.xls to SettlementReport_1-10th.xls

Get multiple files on a local system.

sftp> mget *.xls

Note: As we can see by default with get command download file in local system with same name. We can download remote file with different name specifying the name at the end. (This applies only while downloading single file).

7. Switching Directories

Switching from one directory to another directory in local and remote locations.

On Remote
sftp> cd test
sftp>
On Local
sftp> lcd Documents

8. Create Directories

Creating new directories on local and remote locations.

sftp> mkdir test
sftp> lmkdir Documents

9. Remove Directories

Remove directory or file in remote system.

sftp> rm Report.xls
sftp> rmdir sub1

Note: To remove/delete any directory from remote location, the directory must be empty.

10. Exist sFTP Shell

The ‘!‘ command drop us in local shell from where we can execute Linux commands. Type ‘exit‘ command where we can see sftp> prompt return.

sftp> !

[root@sftp ~]# exit
Shell exited with status 1
sftp>

Linux : How to split a big file in multiple small files

 

If you are splitting a Text file and want to split it by lines you can do this:

split -l 1000 allRecords.txt new

Which will split the text file in output files of 1000 lines each. This is another way to split a file and is mostly used for text files like logs, sql dumps, csv files, etc..

➜ prodFixAuditLog wc allRecords
901 914 66072 allRecords
➜ split -l 100 allRecords new
➜  ls -l  (will list multiple file with name new)
allRecords newac newaf newai
newaa newad newag newaj
newab newae newah

If you want to combine them back you can use below command

➜ cat new* > combineAllRecords.txt
This will combine all files starting with new into a big file with name combineAllRecords.txt

Linux find command with all possible options

Linux find command

The Linux find command is a very useful and handy command to search for files from the command line. It can be used to find files based on various search criterias like permissions, user ownership, modification date/time, size etc. In this post we shall learn to use the find command along with various options that it supports.

The find command is available on most linux distros by default so you do not have to install any package. The find command is an essential one to learn, if you want to get super productive with the command line on linux.

The basic syntax of the find command looks like this

$ find location comparison-criteria search-term

Basic examples

1. List all files in current and sub directories

This command lists out all the files in the current directory as well as the subdirectories in the current directory.

$ find
.
./abc.txt
./subdir
./subdir/how.php
./cool.php

The command is same as the following

$ find .
$ find . -print

2. Search specific directory or path

The following command will look for files in the test directory in the current directory. Lists out all files by default.

$ find ./test
./test
./test/abc.txt
./test/subdir
./test/subdir/how.php
./test/cool.php

The following command searches for files by their name.

$ find ./test -name "abc.txt"
./test/abc.txt

We can also use wildcards

$ find ./test -name "*.php"
./test/subdir/how.php
./test/cool.php

Note that all sub directories are searched recursively. So this is a very powerful way to find all files of a given extension.

Trying to search the “/” directory which is the root, would search the entire file system including mounted devices and network storage devices. So be careful. Of course you can press Ctrl + c anytime to stop the command.

When specifying the directory ("./test" in this example), its fine to omit the trailing slash. However, if the directory is actually a symlink to some other location then you MUST specify the trailing slash for it to work properly (find ./test/ ...)

Ignore the case

It is often useful to ignore the case when searching for file names. To ignore the case, just use the “iname” option instead of the “name” option.

$ find ./test -iname "*.Php"
./test/subdir/how.php
./test/cool.php
Its always better to wrap the search term (name parameter) in double or single quotes. Not doing so will seem to work sometimes and give strange results at other times.

3. Limit depth of directory traversal

The find command by default travels down the entire directory tree recursively, which is time and resource consuming. However the depth of directory travesal can be specified. For example we don’t want to go more than 2 or 3 levels down in the sub directories. This is done using the maxdepth option.

$ find ./test -maxdepth 2 -name "*.php"
./test/subdir/how.php
./test/cool.php

$ find ./test -maxdepth 1 -name *.php
./test/cool.php

The second example uses maxdepth of 1, which means it will not go lower than 1 level deep, either only in the current directory.

This is very useful when we want to do a limited search only in the current directory or max 1 level deep sub directories and not the entire directory tree which would take more time.

Just like maxdepth there is an option called mindepth which does what the name suggests, that is, it will go atleast N level deep before searching for the files.

4. Invert match

It is also possible to search for files that do no match a given name or pattern. This is helpful when we know which files to exclude from the search.

$ find ./test -not -name "*.php"
./test
./test/abc.txt
./test/subdir

So in the above example we found all files that do not have the extension of php, either non-php files. The find command also supports the exclamation mark inplace of not.

find ./test ! -name "*.php"

5. Combine multiple search criterias

It is possible to use multiple criterias when specifying name and inverting. For example

$ find ./test -name 'abc*' ! -name '*.php'
./test/abc.txt
./test/abc

The above find command looks for files that begin with abc in their names and do not have a php extension. This is an example of how powerful search expressions can be build with the find command.

OR operator

When using multiple name criterias, the find command would combine them with AND operator, which means that only those files which satisfy all criterias will be matched. However if we need to perform an OR based matching then the find command has the “o” switch.

$ find -name '*.php' -o -name '*.txt'
./abc.txt
./subdir/how.php
./abc.php
./cool.php

The above command search for files ending in either the php extension or the txt extension.

6. Search only files or only directories

Sometimes we want to find only files or only directories with a given name. Find can do this easily as well.

$ find ./test -name abc*
./test/abc.txt
./test/abc

Only files

$ find ./test -type f -name "abc*"
./test/abc.txt

Only directories

$ find ./test -type d -name "abc*"
./test/abc

Quite useful and handy!

7. Search multiple directories together

So lets say you want to search inside 2 separate directories. Again, the command is very simple

$ find ./test ./dir2 -type f -name "abc*"
./test/abc.txt
./dir2/abcdefg.txt

Check, that it listed files from 2 separate directories.

8. Find hidden files

Hidden files on linux begin with a period. So its easy to mention that in the name criteria and list all hidden files.

$ find ~ -type f -name ".*"

Find files based on permissions

9. Find files with certain permissions

The find command can be used to find files with a specific permission using the “perm” option. The following command searches for files with the permission 0664

$ find . -type f -perm 0664
./abc.txt
./subdir/how.php
./abc.php
./cool.php

This can be useful to find files with wrong permissions which can lead to security issues. Inversion can also be applied to permission checking.

$ find . -type f ! -perm 0777
./abc.txt
./subdir/how.php
./abc.php
./cool.php

10. Find files with sgid/suid bits set

The “perm” option of find command accepts the same mode string like chmod. The following command finds all files with permission 644 and sgid bit set.

# find / -perm 2644

Similarly use 1664 for sticky bit. The perm option also supports using an alternative syntax instead of octal numbers.

$ find / -maxdepth 2 -perm /u=s 2>/dev/null
/bin/mount
/bin/su
/bin/ping6
/bin/fusermount
/bin/ping
/bin/umount
/sbin/mount.ecryptfs_private

Note that the “2>/dev/null” removes those entries that have an error of “Permission Denied”

11. Find readonly files

Find all Read Only files.

$ find /etc -maxdepth 1 -perm /u=r
/etc
/etc/thunderbird
/etc/brltty
/etc/dkms
/etc/phpmyadmin
... output truncated ...

12. Find executable files

The following command will find executable files

$ find /bin -maxdepth 2 -perm /a=x
/bin
/bin/preseed_command
/bin/mount
/bin/zfgrep
/bin/tempfile
... output truncated ...

Search Files Based On Owners and Groups

13. Find files belonging to particular user

To find all or single file called tecmint.txt under /root directory of owner root.

$ find . -user bob
.
./abc.txt
./abc
./subdir
./subdir/how.php
./abc.php

We could also specify the name of the file or any name related criteria along with user criteria

$ find . -user bob -name '*.php'

Its very easy to see, how we can build up criteria after criteria to narrow down our search for matching files.

14. Search files belonging to group

Find all files that belong to a particular group.

# find /var/www -group developer

Did you know you could search your home directory by using the ~ symbol ?

$ find ~ -name "hidden.php"

Easy!!

Search file and directories based on modification date and time

Another great search criteria that the find command supports is modification and accessed date/times. This is very handy when we want to find out which files were modified as a certain time or date range. Lets take a few examples

15. Find files modified N days back

To find all the files which are modified 50 days back.

# find / -mtime 50

16. Find files accessed in last N days

Find all files that were accessed in the last 50 days.

# find / -atime 50

17. Find files modified in a range of days

Find all files that were modified between 50 to 100 days ago.

# find / -mtime +50 –mtime -100

18. Find files changed in last N minutes.

Find files modified within the last 1 hour.

$ find /home/bob -cmin -60

19. Files modified in last hour

To find all the files which are modified in last 1 hour.

# find / -mmin -60

20. Find Accessed Files in Last 1 Hour

To find all the files which are accessed in last 1 hour.

# find / -amin -60

Search files and directories based on size

21. Find files of given size

To find all 50MB files, use.

# find / -size 50M

22. Find files in a size range

To find all the files which are greater than 50MB and less than 100MB.

$ find / -size +50M -size -100M

23. Find largest and smallest files

The find command when used in combination with the ls and sort command can be used to list out the largest files.
The following command will display the 5 largest file in the current directory and its subdirectory. This may take a while to execute depending on the total number of files the command has to process.

$ find . -type f -exec ls -s {} \; | sort -n -r | head -5

Similary when sorted in ascending order, it would show the smallest files first

$ find . -type f -exec ls -s {} \; | sort -n | head -5

24. Find empty files and directories

The following command uses the “empty” option of the find command, which finds all files that are empty.

# find /tmp -type f -empty

To file all empty directories use the type “d”.

$ find ~/ -type d -empty

Really very simple and easy

Some advanced operations

The find command not only finds files based on a certain criteria, it can also act upon those files using any linux command. For example, we might want to delete some files.

Here are some quick examples

25. List out the found files

Lets say we found files using find command, and now want to list them out as the ls command would have done. This is very easy.

$ find . -exec ls -ld {} \;
drwxrwxr-x 4 enlightened enlightened 4096 Aug 11 19:01 .
-rw-rw-r-- 1 enlightened enlightened 0 Aug 11 16:25 ./abc.txt
drwxrwxr-x 2 enlightened enlightened 4096 Aug 11 16:48 ./abc
drwxrwxr-x 2 enlightened enlightened 4096 Aug 11 16:26 ./subdir
-rw-rw-r-- 1 enlightened enlightened 0 Aug 11 16:26 ./subdir/how.php
-rw-rw-r-- 1 enlightened enlightened 29 Aug 11 19:13 ./abc.php
-rw-rw-r-- 1 enlightened enlightened 0 Aug 11 16:25 ./cool.php

26. Delete all matching files or directories

The following command will remove all text files in the tmp directory.

$ find /tmp -type f -name "*.txt" -exec rm -f {} \;

The same operating can be carried out with directories, just put type d, instead of type f.

Lets take another example where we want to delete files larger than 100MB

$ find /home/bob/dir -type f -name *.log -size +10M -exec rm -f {} \;

Free online web based Terminals and IDEs on cloud

Whenever we want to learn some thing, there is a fight to setup machine, environment etc before we can begin, thanks to new cloud technology now you can start instantly using free online web based Terminals and IDEs on cloud

http://www.tutorialspoint.com/codingground.htm

terminal

 

linuxTerminal

Free Online Linux Terminal to practice commands

Friends,

If you want to try out linux commands without bothering to to through installation then try the free online linux terminal to practice commands.

This is simple

http://bellard.org/jslinux/

linuxTerminal.png

This has some additional features :

http://www.webminal.org/

 

How to create your own linux command ;-)

Hi Friends,

Linux gives us lot of general commands and we have to use a lot many commands to finish a specific task. What if we can create our own command and fire them to do our specific task / repetitive task.

You care create alisa which can contain set of commands. For example each time when I have to check if mongo is running on my box I have to type a big command, how about being creative a publish command with name psm => which means it will ps all process and will figure out the mongo process 😉

alias psm=”ps -eaf | grep mongo”

syntax : alias commandName=“cmd1 ; cmd 2 ; cmd 3”

To persist alais as command forever, save that  in .bash_profile
and reload it using command source ~/.bash_profile

# create your own command for session 

> unalias prem => will remove the alias for that session, you might want to remove from .bash_profile to remove it permanently.