Re: Categorizing multiple texts with TM_CATEGORIZE_KNN in one call

Hi Florian,

Firstly, thanks for your suggestion and sorry for late reply. I changed my solution as you suggested. I created an outer procedure (KNN_CHURN_TEST_OUTER) which reads unlabelled records from a table, and an inner procedure (KNN_CHURN_TEST_INNER) which makes predictions for a record (I have two labels, so it makes two predictions for each record). For each record, I call inner procedure from outer procedure. And yes, it made the things much and much faster when I compare it to the previous version. I will post my procedures at the end of this post, maybe somebody may use it later on.

But I wanna ask that is this still fast enough? I have a train (labelled) table which has 24600 records. It has text, maincategory, subcategory columns where maincategory and subcategory are my labels. Additionally, I have another table which has 8095 records to be predicted for each label. When I start the process, it takes about 140 seconds to finish all predictions for 8095 record (both for two labels) and insert the results into one final table. What will happen, when I have 8 million records to be predicted (assuming the train table size will remain same -actually it may increase as well-)? Will it take 140000 seconds which is about 38 hours? Is that normal or is there a way to increase the speed of the process?

Note: I am using aws r3.2xlarge instance type which has 8 cores, 61GB memory, 1x160 GB SSD. Version is 1.00.110.00.1447753075

Here is the outer and inner procedures.

CREATE PROCEDURE "SYSTEM"."KNN_CHURN_TEST_OUTER" ()    LANGUAGE SQLSCRIPT    AS
BEGIN
/*****************************    Write your procedure logic
*****************************/
DECLARE new_text NCLOB;
 DECLARE id INT;
 DECLARE CURSOR c_products FOR
 SELECT "id","text_data"         FROM "SYSTEM"."AVEA_CHURN_TABLE_TEST";
FOR cur_row as c_products DO  new_text := cur_row."text_data";  id := cur_row."id";    call "SYSTEM"."KNN_CHURN_TEST_INNER" (id, new_text);
 END FOR; 
END;

CREATE PROCEDURE "SYSTEM"."KNN_CHURN_TEST_INNER" (IN id INT, IN new_text nclob)    LANGUAGE SQLSCRIPT    AS
BEGIN
/*****************************    Write your procedure logic
*****************************/
DECLARE sub_cat NVARCHAR(128);
DECLARE main_cat NVARCHAR(128);
DECLARE num INT := 0;
DECLARE num2 INT := 0;
 DECLARE CURSOR c_products FOR
SELECT T.CATEGORY_VALUE, T.NEIGHBOR_COUNT, T.SCORE  FROM TM_CATEGORIZE_KNN(    DOCUMENT :new_text      MIME TYPE 'text/plain'    SEARCH NEAREST NEIGHBORS 22 "text"      FROM "SYSTEM"."aveaLabelledData"    RETURN top 1      "main_category"      from "SYSTEM"."aveaLabelledData"     ) AS T;
 DECLARE CURSOR c_products2 FOR
SELECT T.CATEGORY_VALUE, T.NEIGHBOR_COUNT, T.SCORE  FROM TM_CATEGORIZE_KNN(    DOCUMENT :new_text      MIME TYPE 'text/plain'    SEARCH NEAREST NEIGHBORS 22 "text"      FROM "SYSTEM"."aveaLabelledData"    RETURN top 1      "sub_category"      from "SYSTEM"."aveaLabelledData"     ) AS T;
open c_products;
 begin   FOR cur_row as c_products DO  main_cat := cur_row."CATEGORY_VALUE";      num := num + 1;      END FOR;   IF :num = 0  THEN  main_cat := 'unknown';   END IF;
 end;
close c_products;
open c_products2;
 begin
FOR cur_row2 as c_products2 DO  sub_cat := cur_row2."CATEGORY_VALUE";    num2 := num2 + 1;  
 END FOR;
IF :num2 = 0  THEN  sub_cat := 'unknown';
 END IF;
 end;
close c_products2;
insert into "SYSTEM"."KNN_RESULTS" values (:id, :new_text, main_cat, sub_cat);
 commit;
END;