Sunday, June 26, 2011

PLAYING WITH ORACLE LOBs

LOBs deal with unstructured data, the ones which are the most difficult to store and retrieve in a relational database.

In this article, I am going to discuss extensively and manipulate the ORACLE


  • LOBs that are stored in the database itself like BLOB,CLOB,NCLOB ,and


  • LOBs like BFILE which are Stored outside the database as Operating System files.
First let us create a directory :

create or replace directory my_dir as 'D:\pics';
Directory created.
BFILE:
BFILEs act as a pointer and store the location of the external OS files in database tables.

1. POPULATING BFILE COLUMNS IN DATABASE TABLES:
Now we insert a row in it.

Insert into lob_test values (1,’dwaipayan’,bfilename(‘MY_DIR’,’ pic1.jpg’));

This bfilename function returns a BFILE locator for a physical LOB binary file.
Now, if we wish to display the contents of the table:

select * from lob_test;
SP2-0678: Column or attribute type can not be displayed by SQL*Plus
We get an error as we cannot display the contents of a bfile like this.

Now, let us insert another row in the table:

insert into lob_test values (2,'shiba',bfilename('MY_DIR','9.jpg'));
1 row created.
Using the DBMS_LOB.GETLENGTH procedure we can get the lengths of the OS files as follows:

select dbms_lob.getlength(PIC) from lob_test;
DBMS_LOB.GETLENGTH(PIC)
---------------------------
180496
15032


Note: PIC is a BFILE column

2. Locating the files in OS from a BFILE:

Now, say I have loaded the tables with data, and I come back after 3 months and I don’t remember the locations of the OS files with which I had loaded the data into the tables.
In that case what should I do…
Simply, we take the help of DBMS_LOB.FILEGETNAME procedure.
CREATE or replace FUNCTION get_dir_name (bf BFILE) RETURN VARCHAR2 IS
DIR_ALIAS VARCHAR2(255);
FILE_NAME VARCHAR2(255);
BEGIN
IF bf is NULL
THEN
RETURN NULL;
ELSE
DBMS_LOB.FILEGETNAME (bf, dir_alias, file_name);
RETURN FILE_NAME;
END IF;
END;

select get_dir_name(pic) from lob_test;

GET_DIR_NAME(PIC)
--------------------------------------------------------------------------------
pic1.jpg
9.jpg


3. OPENING A BFILE AND COPYING IT TO ANOTHER OS LOCATION

File type: image (jpg)


CREATE OR REPLACE procedure imagecopy
AS
h1 bfile;
h2 utl_file.file_type;
len integer;
strt integer;
buf raw(32767);
BEGIN
h2 := utl_file.fopen('MY_DIR','copyof9.jpg','Wb');
select pic into h1 from lob_test where empno=1;
dbms_lob.open(h1,dbms_lob.file_readonly);
strt:=1;
select dbms_lob.getlength(h1) into len from dual;
while strt<=len
loop
buf := dbms_lob.substr(h1,10000,strt) ;
utl_file.put_raw(h2,buf,TRUE);
strt := strt+10000;
end loop;
utl_file.fclose(h2);
END;
/

So, the image file that was pointed to by the BFILE in lob_test table with empno=1, is now copied to the new location in the OS under the directory ’MY_DIR’ as 'copyof9.jpg'.
Note: Here we have used utl_file package to open the database file where we are going to write the data. The mode in which the 'copyof9.jpg' file is opened is ‘Wb’ because we are going to write binary data in it.
Utl_file.put_raw is used to write raw data into the file
Besides these, we have used the dbms_lob.substr function.It’s syntax is a bit different from the ordinary substr function. In dbms_lob.substr, the requested substring length comes first, followed by the offset.


BLOB :
We , first create a table test_blob containing a blob column ‘DOC’ :

create table test_blob (empno number,empname varchar2(30),doc blob);

Table created.
1. Populating Blob column in database from OS file:
create or replace procedure blobfromfile
as
blob_buf blob;
h1 bfile;
begin
h1 := bfilename('MY_DIR','9.jpg');
dbms_lob.open(h1,dbms_lob.file_readonly);
blob_buf := dbms_lob.substr(h1,dbms_lob.getlength(h1),1);
insert into test_blob values (1,'dwaipayan',blob_buf);
commit;
end;


select count(*) from test_blob;

COUNT(*)
----------
1



CLOB :
We , first create a table test_clob containing a clob column ‘ABOUT_ME’ :
create table test_clob (empno number,about_me clob);

Table created.

insert into test_clob values (1,empty_clob());

1 row created.
1. Populating Clob column in database from OS file:
CREATE OR REPLACE procedure CLOB_POPULATE
AS
CLOB_BUF blob;
h1 BFILE;
BEGIN
h1 := BFILENAME('MY_DIR','t3.txt');
dbms_lob.open(h1,dbms_lob.file_readonly);
clob_buf := dbms_lob.substr(h1,dbms_lob.getlength(h1),1) ;
update test_clob set about_me=utl_raw.cast_to_varchar2(clob_buf) where empno=1;
commit;
end;

select substr(about_me,1,100) FROM test_clob;

SUBSTR(ABOUT_ME,1,100)
--------------------------------------------------------------------------------
ooo
hhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhh
ooooooooooooooooooooooooooooooooo
hhh
2. CONVERTING CLOB TO BLOB :
insert into test_blob
2 select 2,'sinha',utl_raw.cast_to_raw(about_me) from test_clob where empno=1;

1 row created.

Displaying The raw data:

select dbms_lob.substr(doc,100,1) from test_blob where empno=2;

DBMS_LOB.SUBSTR(DOC,100,1)
--------------------------------------------------------------------------------
0D0A6F6F6F0D0A686868686868686868686868686868686868686868686868686868686868686868
0D0A6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F6F0D0A686868
6868686868686868686868686868686868686868

Monday, June 6, 2011

DBMS_FGA - ORACLE FINE GRAINED AUDITING

Suppose your boss calls you one day and tells you that there has been some unexpected changes in the employee database.Employee's designation, their salary are being manipulated illegally.Such things have been continuing from a past few days and he asks if you could help getting hold of the culprit.
Don't worry Oracle's DBMS_FGA package will save your day and earn you a raise in your job.

The Oracle DBMS_FGA package provides fine grained auditing on objects.


To have an overview of the summary of dbms_fga subprograms visit :

http://download.oracle.com/docs/cd/B19306_01/appdev.102/b14258/d_fga.htm#i1011920

In this article I am going to add a policy on a table FGA_TEST in the SCOTT schema.
The policy will report on any dml actions on this table affecting its 2 columns 'esal' and 'designation'.
Another user HACKER will execute dml queries on this table and we will try and investigate whether the actions of the HACKER are reported.
The corresponding event handler of this policy will be in a 3rd schema FGA_HANDLER .we will also find out if the audit event was handled properly.

Note: Here it is worthwhile mentioning that the DBMS_FGA package can be used not only to audit records in case of data manipulation(DML) but also in cases where data might have been simply viewed depending upon the policy we define. eg:- in case of selecting particular records from a database table.

SQL> show user;
USER is "SCOTT"



First of all let us create a new schema FGA_HANDLER which will contain the event handler .


SQL> create user fga_handler identified by fga_handler;

User created.

SQL> grant resource,connect to fga_handler;

Grant succeeded.



Now, let us create a new table ,FGA_TEST in SCOTT schema, on which we will enforce the audit conditions(policy) with the help of the DBMS_FGA package.


SQL> create table fga_test
2 (empno number,
3 empname varchar2(30),
4 esal number,
5 designation varchar2(20)
6 );

Table created.



Let us insert some dummy rows in it now.


SQL> insert into FGA_TEST
2 SELECT 1,'dwaipayan',20000,'programmer' from dual
3 union all
4 select 2,'dhruva',30000,'analyst' from dual
5 union all
6 select 3,'shiba',40000,'manager' from dual
7 ;

3 rows created.

SQL> select * from fga_test;

EMPNO EMPNAME ESAL DESIGNATION
---------- ------------------------------ ---------- --------------------
1 dwaipayan 20000 programmer
2 dhruva 30000 analyst
3 shiba 40000 manager



Given below are the parameters of the ADD_POLICY Procedure:
Now, let us add a policy on our FGA_TEST table such that whenever any user tries to insert, update or delete the ‘esal’ or the ‘designation’ columns of any row of FGA_TEST table ,the action will be recorded.



BEGIN
DBMS_FGA.ADD_POLICY (
object_schema => 'SCOTT',
object_name => 'FGA_TEST',
policy_name => 'FGA_TEST_POLICY',
audit_condition => NULL,
audit_column => 'ESAL,DESIGNATION',
handler_schema => 'FGA_HANDLER',
handler_module => 'sp_audit',
enable => true,
statement_types => ‘INSERT,UPDATE,DELETE’
);
end;

Note: the default 'statement_types' is 'SELECT'

Now, sp_audit is the audit procedure, which will act as the alerting mechanism for the administrator.

The required interface for such a procedure is as follows:
PROCEDURE ( object_schema VARCHAR2, object_name VARCHAR2, policy_name VARCHAR2 ) AS ....
Now, let us connect to the FGA_HANDLER schema and design the
the sp_audit procedure.

First let us create the table where the sp_audit procedure will dump the data into.


SQL> conn fga_handler/fga_handler;
Connected.

SQL> create table audit_event
2 (audit_event_no number);

Table created.


We create the sp_audit procedure as follows:



SQL> create or replace procedure sp_audit
(object_schema in varchar2,
object_name in varchar2,
policy_name in varchar2
)
as
count number;
begin

select nvl(max(audit_event_no),0) into count from audit_event;

insert into audit_event
values
(count+1);
commit;

end;


The procedure simply adds a record to the audit_event table each time it is executed and the column audit_event_no acts as counter which displays the number of times the proc has been executed.

Now, finally we create another schema ‘HACKER’ which tries to manipulate the values of the ‘esal’ or ‘designation’ columns of the FGA_TEST table.


SQL> create user hacker identified by hacker;
User created.

SQL> grant resource,connect to hacker;
Grant succeeded.

SQL> grant all on fga_test to hacker;
Grant succeeded.
Now we log onto HACKER and try to change the ‘designation’ of ‘dwaipayan’ from ‘Programmer’ to ‘manager’

SQL> conn hacker/hacker;
Connected.

SQL> update scott.fga_test set designation='manager' where empname='dwaipayan';

1 row updated.



Now, we connect with SCOTT to see the dba_fga_audit_trail view to find if the event was recorded.


SQL> select DB_USER,OS_USER,POLICY_NAME,SQL_TEXT,
TIMESTAMP from dba_fga_audit_trail
where POLICY_NAME='FGA_TEST_POLICY';







DB_USER OS_USER POLICY_NAME SQL_TEXT TIMESTAMP
HACKER HOME-6C286D743C\Dwaipayan FGA_TEST_POLICY update scott.fga_test
set
designation
= 'manager'
where
empname='dwaipayan'
5-Jun-11



Now we connect to the FGA_HANDLER schema to see if the event handler(sp_audit) was called:


SQL> conn fga_HANDLER/fga_handler;
Connected.
SQL> select * from audit_event;

AUDIT_EVENT_NO
--------------
1



We execute the following from HACKER schema:

SQL>update SCOTT.FGA_TEST set designation='HR' where name='shiba';
SQL> conn fga_HANDLER/fga_handler;
Connected.
SQL> select * from audit_event;


AUDIT_EVENT_NO
--------------
1
2



Note: I have just created the sp_audit procedure to add rows to the audit_event table in this testing environment but in a ideal production scenario, we are likely to send emails or Page instead.