12 July, 2020

A Function Based Index : 2

Here's another Function Based Index example.   This is in 19.3  (not to demonstrate any difference between  12.2 and 19.3 here)

SQL> create table employees
  2  (employee_id number primary key,
  3  first_name varchar2(30),
  4  last_name varchar2(30))
  5  pctfree 0
  6  /

Table created.

SQL>SQL> insert into employees
...
...

10000 rows created.

SQL> commit;

Commit complete.

SQL>
SQL> exec dbms_stats.gather_table_stats('','EMPLOYEES');

PL/SQL procedure successfully completed.

SQL>
SQL> select sample_size, blocks, empty_blocks, avg_row_len
  2  from user_tab_statistics
  3  where table_name  = 'EMPLOYEES'
  4  /

SAMPLE_SIZE     BLOCKS EMPTY_BLOCKS AVG_ROW_LEN
----------- ---------- ------------ -----------
      10000         65            0          45

SQL>
SQL> select 45*10000/8192 from dual;

45*10000/8192
-------------
   54.9316406

SQL>
SQL> select column_name, avg_col_len
  2  from user_tab_cols
  3  where table_name = 'EMPLOYEES'
  4  order by column_id
  5  /

COLUMN_NAME                    AVG_COL_LEN
------------------------------ -----------
EMPLOYEE_ID                              4
FIRST_NAME                              19
LAST_NAME                               22

SQL>
SQL> create index employees_full_name_ndx
  2  on employees(first_name||last_name)
  3  /

Index created.

SQL>
SQL> select column_name, avg_col_len
  2  from user_tab_cols
  3  where table_name = 'EMPLOYEES'
  4  order by column_id
  5  /

COLUMN_NAME                    AVG_COL_LEN
------------------------------ -----------
EMPLOYEE_ID                              4
FIRST_NAME                              19
LAST_NAME                               22
SYS_NC00004$

SQL>
SQL> -- no statistics on the new virtual column yet
SQL> exec dbms_stats.gather_table_stats('','EMPLOYEES');

PL/SQL procedure successfully completed.

SQL> select column_name, avg_col_len
  2  from user_tab_cols
  3  where table_name = 'EMPLOYEES'
  4  order by column_id --- for the hidden/virtual column actually use "internal_column_id"
  5  /

COLUMN_NAME                    AVG_COL_LEN
------------------------------ -----------
EMPLOYEE_ID                              4
FIRST_NAME                              19
LAST_NAME                               22
SYS_NC00004$                            40

SQL>
SQL> select data_type, data_length,data_default,column_id,internal_column_id, hidden_column,virtual_column
  2  from user_tab_cols
  3  where table_name = 'EMPLOYEES'
  4* and column_name = 'SYS_NC00004$'
SQL> /

DATA_TYPE        DATA_LENGTH DATA_DEFAULT                COLUMN_ID INTERNAL_COLUMN_ID HID VIR
---------------- ----------- -------------------------- ---------- ------------------ --- ---
VARCHAR2                  60 "FIRST_NAME"||"LAST_NAME"                              4 YES YES

SQL>
SQL> select num_rows, sample_size, blocks, empty_blocks, avg_row_len
  2  from user_tab_statistics
  3  where table_name  = 'EMPLOYEES'
  4  /

  NUM_ROWS SAMPLE_SIZE     BLOCKS EMPTY_BLOCKS AVG_ROW_LEN
---------- ----------- ---------- ------------ -----------
     10000       10000         65            0          85

SQL>
SQL> select 85*10000/8192 from dual;

85*10000/8192
-------------
   103.759766

SQL>
SQL> select leaf_blocks
  2  from user_ind_statistics
  3  where index_name = 'EMPLOYEES_FULL_NAME_NDX'
  4  /

LEAF_BLOCKS
-----------
         72

SQL>



After creation of the Function Based Index, the new virtual column in the table has an average length of 40 bytes.  Surely, the table should now have taken up more blocks with the average row length increasing from 45 to 85 ?  The number of table blocks should have increased by about 50 or more blocks ? And, yet, the table still consumes only 65 blocks.  It is the Index that reports 72 leaf blocks.

So, adding a Function Based Index increases the reported "AVG_ROW_LEN" for the Table without actually increasing the space consumed by the table.  The computed values are not stored in the Table blocks but are actually only in the Index leaf blocks.

Conversely, this can also mean that when you come across a table with a low BLOCKS versus AVG_ROW_LEN*NUM_ROWS, you might want to look for a Function Based Index :


SQL> select index_name, index_type, uniqueness
  2  from user_indexes
  3  where table_name =  'EMPLOYEES'
  4  /

INDEX_NAME                     INDEX_TYPE                  UNIQUENES
------------------------------ --------------------------- ---------
SYS_C007674                    NORMAL                      UNIQUE
EMPLOYEES_FULL_NAME_NDX        FUNCTION-BASED NORMAL       NONUNIQUE

SQL> select column_position, column_expression
  2  from user_ind_expressions
  3  where table_name =  'EMPLOYEES'
  4  and index_name = 'EMPLOYEES_FULL_NAME_NDX'
  5  /

COLUMN_POSITION COLUMN_EXPRESSION
--------------- ------------------------------
              1 "FIRST_NAME"||"LAST_NAME"

SQL>


(SYS_C007674 is the Primary Key Index on EMPLOYEE_ID)


09 July, 2020

Adding a Function-Based Index

What happens when you create a Function Based Index ?

{This demo is in 12.2}

I start with my "large" table.

SQL> create table my_data_table 
  2  as select * from dba_objects
  3  union all select * from dba_objects;

Table created.

SQL> select count(*) from my_data_table;

  COUNT(*)
----------
    157408

SQL> exec dbms_stats.gather_table_stats('','MY_DATA_TABLE',degree=>4);

PL/SQL procedure successfully completed.

SQL> select num_rows, avg_row_len
  2  from user_tables
  3  where table_name = 'MY_DATA_TABLE'
  4  /

  NUM_ROWS AVG_ROW_LEN
---------- -----------
    157408         132

SQL> 


I then add a Function Based Index on it.

SQL> create index my_data_fbi   
  2  on my_data_table(lower(OBJECT_NAME))
  3  /

Index created.

SQL> 


I now look at Table and Column statistics

SQL> exec dbms_stats.gather_table_stats('','MY_DATA_TABLE',degree=>4);

PL/SQL procedure successfully completed.

SQL> 
SQL> desc my_data_table
 Name                                                                     Null?    Type
 ------------------------------------------------------------------------ -------- -------------------------------------------------
 OWNER                                                                             VARCHAR2(128)
 OBJECT_NAME                                                                       VARCHAR2(128)
 SUBOBJECT_NAME                                                                    VARCHAR2(128)
 OBJECT_ID                                                                         NUMBER
 DATA_OBJECT_ID                                                                    NUMBER
 OBJECT_TYPE                                                                       VARCHAR2(23)
 CREATED                                                                           DATE
 LAST_DDL_TIME                                                                     DATE
 TIMESTAMP                                                                         VARCHAR2(19)
 STATUS                                                                            VARCHAR2(7)
 TEMPORARY                                                                         VARCHAR2(1)
 GENERATED                                                                         VARCHAR2(1)
 SECONDARY                                                                         VARCHAR2(1)
 NAMESPACE                                                                         NUMBER
 EDITION_NAME                                                                      VARCHAR2(128)
 SHARING                                                                           VARCHAR2(18)
 EDITIONABLE                                                                       VARCHAR2(1)
 ORACLE_MAINTAINED                                                                 VARCHAR2(1)
 APPLICATION                                                                       VARCHAR2(1)
 DEFAULT_COLLATION                                                                 VARCHAR2(100)
 DUPLICATED                                                                        VARCHAR2(1)
 SHARDED                                                                           VARCHAR2(1)
 CREATED_APPID                                                                     NUMBER
 CREATED_VSNID                                                                     NUMBER
 MODIFIED_APPID                                                                    NUMBER
 MODIFIED_VSNID                                                                    NUMBER

SQL> 
SQL> l
  1  select column_name, num_nulls,to_char(last_analyzed,'DD-MON HH24:MI')
  2  from user_tab_columns
  3  where table_name =  'MY_DATA_TABLE'
  4* order by column_id
SQL> /

COLUMN_NAME                    NUM_NULLS  TO_CHAR(LAST_ANALYZED
------------------------------ ---------- ---------------------
OWNER                                   0 09-JUL 22:43
OBJECT_NAME                             0 09-JUL 22:43
SUBOBJECT_NAME                     155954 09-JUL 22:43
OBJECT_ID                               2 09-JUL 22:43
DATA_OBJECT_ID                     139120 09-JUL 22:43
OBJECT_TYPE                             0 09-JUL 22:43
CREATED                                 0 09-JUL 22:43
LAST_DDL_TIME                           2 09-JUL 22:43
TIMESTAMP                               2 09-JUL 22:43
STATUS                                  0 09-JUL 22:43
TEMPORARY                               0 09-JUL 22:43
GENERATED                               0 09-JUL 22:43
SECONDARY                               0 09-JUL 22:43
NAMESPACE                               2 09-JUL 22:43
EDITION_NAME                       157408 09-JUL 22:43
SHARING                                 0 09-JUL 22:43
EDITIONABLE                        101970 09-JUL 22:43
ORACLE_MAINTAINED                       0 09-JUL 22:43
APPLICATION                             0 09-JUL 22:43
DEFAULT_COLLATION                  125494 09-JUL 22:43
DUPLICATED                              0 09-JUL 22:43
SHARDED                                 0 09-JUL 22:43
CREATED_APPID                      157408 09-JUL 22:43
CREATED_VSNID                      157408 09-JUL 22:43
MODIFIED_APPID                     157408 09-JUL 22:43
MODIFIED_VSNID                     157408 09-JUL 22:43

26 rows selected.

SQL> 
SQL> l
  1  select column_name, num_nulls,to_char(last_analyzed,'DD-MON HH24:MI')
  2  from user_tab_cols
  3  where table_name =  'MY_DATA_TABLE'
  4* order by column_id
SQL> /

COLUMN_NAME                    NUM_NULLS  TO_CHAR(LAST_ANALYZED
------------------------------ ---------- ---------------------
OWNER                                   0 09-JUL 22:43
OBJECT_NAME                             0 09-JUL 22:43
SUBOBJECT_NAME                     155954 09-JUL 22:43
OBJECT_ID                               2 09-JUL 22:43
DATA_OBJECT_ID                     139120 09-JUL 22:43
OBJECT_TYPE                             0 09-JUL 22:43
CREATED                                 0 09-JUL 22:43
LAST_DDL_TIME                           2 09-JUL 22:43
TIMESTAMP                               2 09-JUL 22:43
STATUS                                  0 09-JUL 22:43
TEMPORARY                               0 09-JUL 22:43
GENERATED                               0 09-JUL 22:43
SECONDARY                               0 09-JUL 22:43
NAMESPACE                               2 09-JUL 22:43
EDITION_NAME                       157408 09-JUL 22:43
SHARING                                 0 09-JUL 22:43
EDITIONABLE                        101970 09-JUL 22:43
ORACLE_MAINTAINED                       0 09-JUL 22:43
APPLICATION                             0 09-JUL 22:43
DEFAULT_COLLATION                  125494 09-JUL 22:43
DUPLICATED                              0 09-JUL 22:43
SHARDED                                 0 09-JUL 22:43
CREATED_APPID                      157408 09-JUL 22:43
CREATED_VSNID                      157408 09-JUL 22:43
MODIFIED_APPID                     157408 09-JUL 22:43
MODIFIED_VSNID                     157408 09-JUL 22:43
SYS_NC00027$                            0 09-JUL 22:43

27 rows selected.

SQL> 

SQL> l
  1  select column_name, avg_col_len
  2  from user_tab_cols
  3  where table_name = 'MY_DATA_TABLE'
  4* order by column_id
SQL> /

COLUMN_NAME                    AVG_COL_LEN
------------------------------ -----------
OWNER                                    6
OBJECT_NAME                             34
SUBOBJECT_NAME                           2
OBJECT_ID                                5
DATA_OBJECT_ID                           2
OBJECT_TYPE                             10
CREATED                                  8
LAST_DDL_TIME                            8
TIMESTAMP                               20
STATUS                                   7
TEMPORARY                                2
GENERATED                                2
SECONDARY                                2
NAMESPACE                                3
EDITION_NAME                             0
SHARING                                 13
EDITIONABLE                              2
ORACLE_MAINTAINED                        2
APPLICATION                              2
DEFAULT_COLLATION                        4
DUPLICATED                               2
SHARDED                                  2
CREATED_APPID                            0
CREATED_VSNID                            0
MODIFIED_APPID                           0
MODIFIED_VSNID                           0
SYS_NC00027$                            34

27 rows selected.

SQL> 
SQL> select num_rows, avg_row_len
  2  from user_tables
  3  where table_name = 'MY_DATA_TABLE'
  4  /

  NUM_ROWS AVG_ROW_LEN
---------- -----------
    157408         166

SQL> 


A new column "SYS_NC00027$" appears in USER_TAB_COLS but not in USER_TAB_COLUMNS.  The new column is not also not visible when I run a "DESCRIBE" command.

Also, the AVG_ROW_LEN has increased by 34  (seeing as I do not have any objects with long names) to reflect the addition of the new virtual column.
But did all the blocks actually get re-written ?  Are the "lower(OBJECT_NAME)" values written into each table block as well ?

SQL> exec dbms_stats.gather_table_stats('','MY_DATA_TABLE',degree=>4);

PL/SQL procedure successfully completed.

SQL> 
SQL> l
  1  select obj#, savtime, rowcnt, blkcnt, avgrln,samplesize,to_char(analyzetime,'DD-MON-RR HH24:MI:SS')
  2  from sys.WRI$_OPTSTAT_TAB_HISTORY
  3  where obj# =
  4  (select object_id
  5   from dba_objects
  6   where owner = 'HEMANT'
  7   and object_type = 'TABLE'
  8   and object_name = 'MY_DATA_TABLE')
  9* order by 1,2
SQL> /

      OBJ# SAVTIME                      ROWCNT     BLKCNT     AVGRLN     SAMPLESIZE TO_CHAR(ANALYZETIME,'DD-MON
---------- ---------------------------- ---------- ---------- ---------- ---------- ---------------------------
     79843 09-JUL-20 10.39.03.789392 PM     157408       3106        132     157408 09-JUL-20 22:37:53
     79843 09-JUL-20 10.43.59.424420 PM     157408       3106        132     157408 09-JUL-20 22:39:03
     79843 09-JUL-20 11.02.35.088733 PM     157408       3106        166     157408 09-JUL-20 22:43:59

SQL> 


Apparently, the actual number of blocks did not increase.

SQL> l
  1  select segment_name, segment_type, bytes/1024
  2  from user_segments
  3  where segment_name like 'MY_DATA%'
  4* order by 1,2
SQL> /

SEGMENT_NAME                   SEGMENT_TYPE       BYTES/1024
------------------------------ ------------------ ----------
MY_DATA_FBI                    INDEX                    8192
MY_DATA_TABLE                  TABLE                   25600

SQL> 
SQL> 
SQL> l
  1  select index_name, column_name
  2  from user_ind_columns
  3* where index_name = 'MY_DATA_FBI'
SQL> /

INDEX_NAME                     COLUMN_NAME
------------------------------ ------------------------------
MY_DATA_FBI                    SYS_NC00027$

SQL> 


So, the Index does take up space as an Index segment but the Virtual Column ("lower(OBJECT_NAME)")  created on the Table does not grow the table.  What seems to happen is that the "computed" value ("lower(OBJECT_NAME)") is stored in the Index segment (leaf blocks) but not in the Table segment (blocks)


What happens if I drop the Function Based Index ?

SQL> drop index my_data_fbi;

Index dropped.

SQL> exec dbms_stats.gather_table_stats('','MY_DATA_TABLE',degree=>4);

PL/SQL procedure successfully completed.

SQL> 
SQL> select count(*)
  2  from user_tab_columns
  3  where table_name = 'MY_DATA_TABLE'
  4  /

  COUNT(*)
----------
        26

SQL> select count(*)
  2  from user_tab_cols
  3  where table_name = 'MY_DATA_TABLE'
  4  /

  COUNT(*)
----------
        26

SQL> select avg_row_len
  2  from user_tables
  3  where table_name = 'MY_DATA_TABLE'
  4  /

AVG_ROW_LEN
-----------
        132

SQL> 


The additional column and the 34 bytes it represents has disappeared from the Table and Column statistics.



07 July, 2020

A Correlated SubQuery

I take this simple Correlated Subquery that identifies all employees whose Salary is greater than the department's average Salary and is also greater than $9,000

SQL> l
  1  select emp.department_id, emp.employee_id, emp.salary
  2  from hr.employees emp
  3  where emp.salary >
  4  (select avg(salary)
  5   from hr.employees emp_inner
  6   where emp.department_id = emp_inner.department_id)
  7  and emp.salary > 9000
  8* order by 1,2
SQL> /

DEPARTMENT_ID EMPLOYEE_ID     SALARY
------------- ----------- ----------
           20         201      13000
           30         114      11000
           80         145      14000
           80         146      13500
           80         147      12000
           80         148      11000
           80         149      10500
           80         150      10000
           80         151       9500
           80         156      10000
           80         157       9500
           80         162      10500
           80         163       9500
           80         168      11500
           80         169      10000
           80         170       9600
           80         174      11000
           90         100      24000
          100         108      12008
          110         205      12008

20 rows selected.

SQL>


The Row Source Statistics for the actual query execution are :

select emp.department_id, emp.employee_id, emp.salary
from hr.employees emp
where emp.salary >
(select avg(salary)
 from hr.employees emp_inner
 where emp.department_id = emp_inner.department_id)
and emp.salary > 9000
order by 1,2

call     count       cpu    elapsed       disk      query    current        rows
------- ------  -------- ---------- ---------- ---------- ----------  ----------
Parse        1      0.00       0.00          0          0          0           0
Execute      1      0.00       0.00          0          0          0           0
Fetch        3      0.00       0.00          0         12          0          20
------- ------  -------- ---------- ---------- ---------- ----------  ----------
total        5      0.00       0.00          0         12          0          20

Misses in library cache during parse: 1
Optimizer mode: ALL_ROWS
Parsing user id: 106
Number of plan statistics captured: 1

Rows (1st) Rows (avg) Rows (max)  Row Source Operation
---------- ---------- ----------  ---------------------------------------------------
        20         20         20  FILTER  (cr=12 pr=0 pw=0 time=1027 us starts=1)
        23         23         23   SORT GROUP BY (cr=12 pr=0 pw=0 time=1020 us starts=1 cost=7 size=3552 card=111)
       536        536        536    HASH JOIN  (cr=12 pr=0 pw=0 time=917 us starts=1 cost=6 size=71648 card=2239)
        23         23         23     TABLE ACCESS FULL EMPLOYEES (cr=6 pr=0 pw=0 time=30 us starts=1 cost=3 size=1752 card=73)
       108        108        108     TABLE ACCESS FULL EMPLOYEES (cr=6 pr=0 pw=0 time=7 us starts=1 cost=3 size=864 card=108)

********************************************************************************


The Execution Plan is

----------------------------------------------------------------------------------
| Id  | Operation            | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------------
|   0 | SELECT STATEMENT     |           |   111 |  3552 |     7  (15)| 00:00:01 |
|*  1 |  FILTER              |           |       |       |            |          |
|   2 |   SORT GROUP BY      |           |   111 |  3552 |     7  (15)| 00:00:01 |
|*  3 |    HASH JOIN         |           |  2239 | 71648 |     6   (0)| 00:00:01 |
|*  4 |     TABLE ACCESS FULL| EMPLOYEES |    73 |  1752 |     3   (0)| 00:00:01 |
|   5 |     TABLE ACCESS FULL| EMPLOYEES |   108 |   864 |     3   (0)| 00:00:01 |
----------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - filter("EMP"."SALARY">SUM("SALARY")/COUNT("SALARY"))
   3 - access("EMP"."DEPARTMENT_ID"="EMP_INNER"."DEPARTMENT_ID")
   4 - filter("EMP"."SALARY">9000)


(Since I had only 1 execution of the query when tracing is enabled, "Rows (1st"), "Rows (avg)" and "Rows (max)" report on the same, single, execution)

The EMPLOYEES table underwent two Full Table Scans, each consisting of 6 "consistent reads"  ("cr=6"). 

The execution at line Id=4 expected to return 73 rows ("card=73") but actually returned 23 rows.  This query filtered for SALARY greater than 9000.

The execution at line Id=5 expected to return 108 rows ("card=108") and did actually return 108 rows.  There is no predicate information, meaning that all 108 rows of the table were returned, none filtered.

The join of the outer query and the inner query on DEPARTMENT_ID was the Hash Join at operation Id=3.

Do you note how the FILTER at operation ID=1 appears ?  It is filtering for SALARY greater than AVERAGE SALARY  (where AVERAGE is computed as SUM of SALARY  values divided by the number of not-null SALARY values)