Showing posts with label SAS. Show all posts
Showing posts with label SAS. Show all posts

Sunday, March 22, 2015

The 2015 Rexer Analytics Data Miner Survey has launched!

The 2015 Rexer Analytics Data Miner Survey has launched. The founder of Rexer Analytics, Karl Rexer, is regarded by Deep-data-mining.com as one of 10 Most Influential People in Data Analytics.

Analytic Professionals — Share your views: Participate in the 2015 Data Miner Survey

Data Analysts, Predictive Modelers, Data Scientists, Data Miners, and all other types of analytic professionals, students, and academics: Please participate in the 2015 Rexer Analytics Data Miner Survey.


Survey Link: www.rexeranalytics.com/Data-Miner-Survey-2015-Intro.html
Access Code: CL72L4 — It is OK to share this Access Code with others: It can be used by multiple people.
Survey results will be unveiled at the Fall-2015 Boston Predictive Analytics World event.

Rexer Analytics has been conducting the Data Miner Survey since 2007. Each survey explores the analytic behaviors, views and preferences of data miners and analytic professionals. Over 1200 people from around the globe participated in the 2013 survey. Summary reports (40 page PDFs) from previous surveys are available FREE to everyone who requests them by emailing DataMinerSurvey@RexerAnalytics.com. Also, highlights of earlier Data Miner Surveys are available at www.rexeranalytics.com/Data-Miner-Survey-Results-2013.html, including best practices shared by respondents on analytic success measurement, overcoming data mining challenges, and other topics. The FREE Summary Report for this 2015 Data Miner Survey will be available to everyone Fall-2015.

Please tell other data analysis professionals about the survey.

Rexer Analytics is a consulting firm focused on providing data mining and analytic CRM solutions. Recent solutions include customer loyalty analyses, customer segmentation, predictive modeling to predict customer attrition and to target direct marketing, fraud detection, sales forecasting, market basket analyses, and complex survey research. More information is available at www.RexerAnalytics.com or by calling +1 617-233-8185.

Saturday, September 01, 2012

Data Mining Components

We identify four components or layers in a data mining engagement as shown in the figure below. Two abstract layers, business problem and data mining algorithms, are on the top. And two physical layers, data mining tools and data management, are at the bottom.

Business problems that we want to solve is one of the most important abstract layers. It could be predicting fraud (bank card, check, medical claims), new customer life time value at point of sales, online ads click rate, credit worthiness, customer segmentation. We can address the above business problems using various predictive models such as logistic regressions, neural nets, support vector machines, K-means clustering.

Data mining tools layer contains commercial or open source software such as SAS, Splus, R, Weca, SPSS, Statsoft, Oracle Data Mining. Common data mining algorithms can be found in almost all of the software mentioned above. Data management/storage layer are relational databases such as Oracle, SQL server, MySQL, or simply files such as SAS files or text files.

We can predict if a new customer will pay his car loan using logistic regression model implemented in SAS and store the data in Oracle. Or we can solve the same problem using decision tree models implemented in R and store data in SQL server. It is important to realize that items within each layer are sometimes exchangeable. We can solve any business problems with varieties of data mining algorithms implemented by commercial or open source tools and store data in any databases.  Thus it is a misconcept that neural nets are best in predicting credit card fraud. An experienced data miner can build a decision tree model to predict credit card fraud that performs equally well as a neural net does. We can select the combination of data mining models, tools and databases that suit our needs.
Four Components or Layers in Data Mining


Thursday, May 31, 2012

Floating Point Representation: the Loss of Precision

On SAS PC, 11169568236203649 is actually represented as 11169568236203648 due to the limitation of  64-bit IEEE floating point representation. 


In most of the applications, there is practically no difference between  11169568236203649  and 11169568236203648. Unfortunately, every digit matters in the case of function mod. 

  

Wednesday, May 30, 2012

A Clarification on SAS mod function.

I am a long term SAS fan.  Its statistical functions and powerful macro scripting capability are simply unprecedented.  There is a little bit of misunderstanding concerning my post on MOD function.  The following is a clarification from Rick Langston from SAS.

Rick's clarification follows.
--------------------------------------------------------
There is no such "bug" in the MOD function in SAS.
The problem is due to the limitation of IEEE floating point
representation with the number of digits used in the example.
The number 11169568236203649 cannot be precisely stored in a 64-bit
IEEE floating point value. It is indistinguishable from
11169568236203648, for example.
SAS uses floating point representation for all of its numeric values.
Note that if you apply the same constraints to databases, you
will see the same results as SAS. For example, on Teradata
if you submit this expression:

select cast(11169568236203649 as float) mod cast(30269 as float);
You get the response of
        1.17310000000000E 004
Which is just what SAS provides.

And consider this C code:
main()
{
double x;
x = (long long)11169568236203649.0 % (long long)30269.0;
printf("x=%f.\n",x);
x = 11169568236203649LL % 30269LL;
printf("x=%f.\n",x);
}

The results are this:
x=11731.000000.
x=11732.000000.

It's the same issue. The double-precision value cannot
be represented sufficiently. If the value remains as a long
integer, the representation is maintained.

Note also that when you run the same MOD function example on
z/OS (i.e. IBM mainframe), you get 11732. This is because the
IBM floating point representation has 4 more bits of mantissa
(at the expense of 4 less bits of exponent) than does IEEE. So
more digits of precision will be noticed, as is the case with the
example.

It had been commented that 64-bit systems might make a difference,
such as SAS on 64-bit Windows as compared to 32-bit Windows.
That is not the case here. The IEEE floating point representation
used by SAS is 64-bit on all platforms. Only the z/OS implementation
of SAS uses the IBM representation.
-----------------------
End of Rick's clarification

It is not a bug in SAS mod function.

My original suspicion was an issue with SAS mod function itself. It is actually because the number 11,169,568,236,203,649 is larger than the maxim integer9,007,199,254,740,992  supported by SAS PC vesion. It is a not a bug since it is clearly stated in the document that there is a maximum integer without rounding supported by SAS.


I was building random generators on both SAS and Oracle. It is required that given the same initial number, the random generators on SAS and Oracle will produce the same series of pseudo random numbers.  Since it is impossible to match the SAS and Oracle results when the initial number is longer than 16 digits (Oracle integer supports 38 digits of precision), in my code I will throw an exception, stop the process and print a message similar to ORA-01438: "value larger than specified precision allowed for this column". 

Tuesday, May 29, 2012

A Bug in SAS MOD Function.

MOD( m, n)  is one of the most commonly used functions. It  returns the remainder of m divided by n and the result is deterministic. We would expect it to be calculated correctly by any data analysis software packages.

However, we have found an issue in SAS mod function.The calculation we tested is  mod(11169568236203649, 30269).  SAS returns different results than Oracle and R do. SAS returns 11731. Both Oracle and R return 11732.

SAS script (SAS Version 9.2) and the result.
data test; val=mod(11169568236203649, 30269) ; run;
11731
or
proc sql; select mod(11169568236203649, 30269) from dummy;
11731

Oracle SQL script and the result.
select mod(11169568236203649, 30269) from dual;
11732

R script and the result.
11169568236203649 %% 30269
11732

We have verified that 11732 is the correct answer using two independent approaches. You may try the above scripts on your own computer and let me know what you find out. ( I explain my approaches to verify that 11732 is the correct answer of mod(11169568236203649,30269) and why I bother to calculate mod(x, 30269) in other posts on this blog.)

SAS Is Wrong! R and Oracle Are Right. MOD(11169568236203649, 30269)=11732



For function MOD(11169568236203649, 30269), SAS returns 11731 while both Oracle  and R  return 11732. I used the following two approaches to verify SAS Is Wrong.  Oracle and R are right.
Approach 1. The following two Oracle Queries show that 11732 is the remainder
SQL>  select ( 11169568236203649 -11732)/30269 vl from dual;
        369010150193.000000
SQL>  select ( 11169568236203649 -11731)/30269 vl from dual;
        369010150193.000033


Approach 2. The following PL/SQL scripts.
At first, I wanted to run the following PL/SQL Script 1. It took too long and I had to kill the process. Instead, I ran Script 2 shown below and it returned  11732 at the end of the loop.
PL/SQL Script 1.
set serveroutput on;
declare
remainder number;
begin
remainder:=11169568236203649;
while (remainder >30269)
loop
remainder:=remainder-30269;
end loop;
dbms_output.put_line(remainder);
end;
/
PL/SQL Script 2.
set serveroutput on;
declare
remainder number;
begin
remainder:=11169568236203649;
while (remainder>302690000000)
loop
remainder:=remainder-302690000000; /* equivalent of 10,000,000 loops of -30269 */
end loop;
dbms_output.put_line(remainder);
while (remainder>30269)
loop
remainder:=remainder-30269;
end loop;
dbms_output.put_line(remainder);
end;
/
At the end of the loop, it returns 11732.

Saturday, May 26, 2012

SAS vs SQL Database: Select an Enterprise Analytic Platform

Without doubt, SAS has the most sophisticated statistical models. However, as data analyitcs practitioners in real business world, we have realized that 80% or more of the time is spent on data preparation and model deployment. Thus, to decide which analytic platform to use in an company, we need to carefully assess factors including how the models will be used (online or offline), the staff's knowlege levels about SAS and SQL, data security, data size, costs, etc.

In my opinion, data should be stored, managed and analyzed in a SQL based database as much as possible. For example, with Oracle 11g, we can perform all the tasks including data collection, summary, manipulation, model building, model scoring, reporting, within a single environment. This in-database analytics approach greatly increases the productivity, manageability and security.

SAS and SQL (Oracle) Version of Univariate Statistics


The following SAS and SQL scripts produce exactly the same results.

SAS Version

proc univariate data = claims ;
by grp;
var payment_amt;
output out = claims_stats N = lines min = var_min
max = var_max mean = var_mean std = var_stnd_dev;
Run;


SQL Version

create materialized view claims
as select
grp,
count(1) lines,
minpayment_amt) var_min,
max(payment_amt) var_max,
avg(payment_amt) var_mean,
stddev(payment_amt) var_stnd_dec
from claim
group by grp;