Real time Problems in datastage with Solutions

1)I am getting this error while opening the administrator client.

DSR_ParseConfig: OPENSEQ of /opt/IBM/InformationServer/Server/PXEngine/etc/master_config.apt took ELSE clause, STATUS()=2 


The default permission for the etc directory is rwxr-xr-x (755). This means that only the owner has write permission. Update the permission of this etc directory (located under ../IBM/InformationServer/Server/PXEngine on the WebSphere DataStage server machine) to give write permission to the user.
To do this, change the permission to allow group write. In the Unix shell, cd to the PXEngine directory and issue the command:

chmod 775 etc 


2)How can we done the Unit Testing on datastage Jobs?


Usually, Unit testing covers the testing of individual job/sequence.

We need to ensure whether the job is completely performing the task, what it is supposed to do.

It should fullfill all the business logic coded in the job. We should prepare the test cases corresponding to all key logic coded like

- check of all functions coded,
- creation of file/dataset,
- insert/update/delete logic,
- Nullablilty checking etc..

Unit Testing - Need to ensure every single expression/logic is working fine.


3)I need to do multiple lookups for different columns in the source against the same dataset. I cannot use the Range lookup since the values are not necessarily in a certain range. 

Do I have to create multiple datasets for each lookup? Or, are there any better options that I can avoid creating multiple datasets? 

A)You need to use 'multiple lookup' stages not 'multiple datasets'. 

4)

I am creating a Parallel job using Change Capture Stage. SQL Server(source stage), Oracle (target stage) I like change capture to do a insert when there is a insert(working) update when there is a delete (not working) update when there is a update(working) Please let me know how I can achieve this. A) I'm assuming if you say "not working" , the delete records are not coming out of the Change Capture Stage. If this is the case, inside the Change Capture Stage in the Stage Properties Tab please select the option as "FALSE" for Drop Output For Delete.

5) Just want to understand the partitining concepts in PX.

Ex. I have configuration files with 2 nodes . The source file is partitioned on custno.
e,g custno are 1,2,3,4.

So , datawise there are 4 partitions but only 2 logical nodes.

1> How will the data be allocated to the 2 logical nodes.
2> Also, if have a stage variable.As i understand 2 copies will be created.
or number of stage variable will be equal to number of data partitions. 
A) 

In datastage Partioning and nodes is different concepts. The job can be partioned into seven types ,Nodes is reduce the time given for nodes

ETL Fundamentals

DataStage Parallel Processing Architecture Overview

datastage faqs

 1.how to generate even numbers in surrogate or tranformar stage ? 2. how many ways to remove duplicate values?

To generate even numbers using transformer, we can make use 
of stage variables. declare a stage variable say SV1 and 
initialise it to 0. In the derivation of that stage 
variable increment it by 2 which looks like SV1+2. By doing 
this we can generate even numbers.

how many write modes are there in ds ?

1.Append
2.Overwrite
3.Create(Error if exist)
4.Use existing (discard record)
5.Use existing (discard record & schema)

How to stop a job when its status is running?

To stop a running job go to DataStage Director and click the stop button (or Job -> Stop from menu). If it doesn't help go to Job -> Cleanup Resources, select a process with holds a lock and click Logout

If it still doesn't help go to the datastage shell and invoke the following command: ds.tools 
It will open an administration panel. Go to 4.Administer processes/locks , then try invoking one of the clear locks commands (options 7-10).

How to send notifications from Datastage as a text message (sms) to a cell phone

There is a few possible methods of sending sms messages from Datastage. However, there is no easy way to do this directly from Datastage and all methods described below will require some effort. 

The easiest way of doing that from the Datastage standpoint is to configure an SMTP (email) server as a mobile phone gateway. In that case, a Notification Activity can be used to send message with a job log and any desired details. DSSendMail Before-job or After-job subroutine can be also used to send sms messages. 
If configured properly, the recipients email address will have the following format: 600123456@oursmsgateway.com 

If there is no possibility of configuring a mail server to send text messages, you can to work it around by using an external application run directly from the operational system. There is a whole bunch of unix scripts and applications to send sms messages. 
In that solution, you will need to create a batch script which will take care of sending messages and invoke it from Datastage using ExecDOS or ExecSh subroutines passing the required parameters (like phone number and message body). 

About AWK Command in Unix

1. Renaming within the name:
   ls -1 *old* | awk '{print "mv "$1" "$1}' | sed s/old/new/2 | sh
   (although in some cases it will fail, as in file_old_and_old)
   
2. remove only files:
   ls -l * | grep -v drwx | awk '{print "rm "$9}' | sh
   or with awk alone:
   ls -l|awk '$1!~/^drwx/{print $9}'|xargs rm
   Be careful when trying this out in your home directory. We remove files!
   
3. remove only directories
   ls -l | grep '^d' | awk '{print "rm -r "$9}' | sh
   or
   ls -p | grep /$ | wk '{print "rm -r "$1}'
   or with awk alone:
   ls -l|awk '$1~/^d.*x/{print $9}'|xargs rm -r
   Be careful when trying this out in your home directory. We remove things!
   
4. killing processes by name (in this example we kill the process called netscape):
   kill `ps auxww | grep netscape | egrep -v grep | awk '{print $2}'`
   or with awk alone:
   ps auxww | awk '$0~/netscape/&&$0!~/awk/{print $2}' |xargs kill 

The 15 Most Important UNIX commands

The 15 Most Important UNIX commands

1. man - show manual for a command, example: man ls hit q to exit the man page.
2. cd - change directory, example: cd /etc/
3. ls - list directory, similar to dir on windows. example: ls /etc, use ls -l /etc to see more detail
4. cp - copy a file or directory, example: cp source dest if you want to copy a      directory use the -R option for recursive: cp -R /source /dest
5. mv - move a file, example: mv source dest
6. rm - remove a file, example: rm somefile to remove a directory you may need the -R option, you can also use the -f option which tells it not to confirm each file: rm -Rf /dir
7. cat - concatenate, or output a file cat /var/log/messages
8. more - outputs one page of a file and pauses. example: more /var/log/messages press q to exit before getting to the bottom. You can also pipe to more | more from other commands, for example ls -l /etc | more
9. scp - secure copy, copies a file over SSH to another server. example:scp /local/file user@host.com:/path/to/save/file
10. tar - tape archiver, tar takes a bunch of files, and munges them into one .tar file, the files are often compressed with the gzip algorithm, and use the .tar.gz extension. to create a tar tar -cf archive.tar /directory, then to extract the archive to the current directory runtar -xf archive.tar to use gzip, just add a z to the options, to create a tar.gz: tar -czf archive.tar.gz /dir to extract it tar -xzf archive.tar.gz
11. grep - pattern matcher, grep takes a regular expression, or to match a simple string you can use fast grep, fgrep failure /var/log/messages, I'm usually just looking for a simple pattern so I tend to use fgrep more than regular grep.
12. find - lists files and directories recursively on a single line, I usually pipe grep into the mix when I use find, eg: find / | fgrep log
13. tail - prints the last few lines of a file, this is handy for checking log files tail /var/log/messages if you need see more lines, use the -noption, tail -n 50 /var/log/messages you can also use the -f option, which will continuously show you the end of the file as things are added to it (very handy for watching logs) tail -f /var/log/messages
14. head - same as tail, but shows the first few lines the file
15. vi - text editor, there are several text editors such as emacs, and nano, but vi is usually installed on any server so its a good one to learn. To edit a file type vi file to edit a line press Esc i then to save changes and exit use Esc wq, or to quit without saving use Esc q!. There are a million other commands, but that will enable you to edit files at a basic level.

UNIX FAQS

How do you change File Access Permissions?

Every file has following attributes:
owner's user ID ( 16 bit integer )
owner's group ID ( 16 bit integer )
File access mode word

'r w x -r w x- r w x'

(user permission-group permission-others permission)
r-read, w-write, x-execute
To change the access mode, we use chmod(filename,mode).
Example 1:
To change mode of myfile to 'rw-rw-r–' (ie. read, write permission for user - read,write permission for group - only read permission for others) we give the args as:
chmod(myfile,0664) .
Each operation is represented by discrete values

'r' is 4
'w' is 2
'x' is 1
Therefore, for 'rw' the value is 6(4+2).

Example 2:
To change mode of myfile to 'rwxr–r–' we give the args as:

chmod(myfile,0744).


How do you find out what’s your shell?
 - echo $SHELL
What’s the command to find out today’s date? 
- date
What’s the command to find out users on the system?
 - who
How do you find out the current directory you’re in? 
- pwd
How do you remove a file?
 - rm

Can anyone explain me the following command? I came across this in one of the shell acripts. sed -e '/^$/d' -e 's/^[ ][ ]*/ /' A)This sed command contains two edits: '/^$/d' this deletes all blank lines 's/^[][]*/ /' for each line, replace all leading spaces with one space.