Showing posts with label R. Show all posts
Showing posts with label R. Show all posts

Sunday, March 22, 2015

The 2015 Rexer Analytics Data Miner Survey has launched!

The 2015 Rexer Analytics Data Miner Survey has launched. The founder of Rexer Analytics, Karl Rexer, is regarded by Deep-data-mining.com as one of 10 Most Influential People in Data Analytics.

Analytic Professionals — Share your views: Participate in the 2015 Data Miner Survey

Data Analysts, Predictive Modelers, Data Scientists, Data Miners, and all other types of analytic professionals, students, and academics: Please participate in the 2015 Rexer Analytics Data Miner Survey.


Survey Link: www.rexeranalytics.com/Data-Miner-Survey-2015-Intro.html
Access Code: CL72L4 — It is OK to share this Access Code with others: It can be used by multiple people.
Survey results will be unveiled at the Fall-2015 Boston Predictive Analytics World event.

Rexer Analytics has been conducting the Data Miner Survey since 2007. Each survey explores the analytic behaviors, views and preferences of data miners and analytic professionals. Over 1200 people from around the globe participated in the 2013 survey. Summary reports (40 page PDFs) from previous surveys are available FREE to everyone who requests them by emailing DataMinerSurvey@RexerAnalytics.com. Also, highlights of earlier Data Miner Surveys are available at www.rexeranalytics.com/Data-Miner-Survey-Results-2013.html, including best practices shared by respondents on analytic success measurement, overcoming data mining challenges, and other topics. The FREE Summary Report for this 2015 Data Miner Survey will be available to everyone Fall-2015.

Please tell other data analysis professionals about the survey.

Rexer Analytics is a consulting firm focused on providing data mining and analytic CRM solutions. Recent solutions include customer loyalty analyses, customer segmentation, predictive modeling to predict customer attrition and to target direct marketing, fraud detection, sales forecasting, market basket analyses, and complex survey research. More information is available at www.RexerAnalytics.com or by calling +1 617-233-8185.

Saturday, September 01, 2012

Data Mining Components

We identify four components or layers in a data mining engagement as shown in the figure below. Two abstract layers, business problem and data mining algorithms, are on the top. And two physical layers, data mining tools and data management, are at the bottom.

Business problems that we want to solve is one of the most important abstract layers. It could be predicting fraud (bank card, check, medical claims), new customer life time value at point of sales, online ads click rate, credit worthiness, customer segmentation. We can address the above business problems using various predictive models such as logistic regressions, neural nets, support vector machines, K-means clustering.

Data mining tools layer contains commercial or open source software such as SAS, Splus, R, Weca, SPSS, Statsoft, Oracle Data Mining. Common data mining algorithms can be found in almost all of the software mentioned above. Data management/storage layer are relational databases such as Oracle, SQL server, MySQL, or simply files such as SAS files or text files.

We can predict if a new customer will pay his car loan using logistic regression model implemented in SAS and store the data in Oracle. Or we can solve the same problem using decision tree models implemented in R and store data in SQL server. It is important to realize that items within each layer are sometimes exchangeable. We can solve any business problems with varieties of data mining algorithms implemented by commercial or open source tools and store data in any databases.  Thus it is a misconcept that neural nets are best in predicting credit card fraud. An experienced data miner can build a decision tree model to predict credit card fraud that performs equally well as a neural net does. We can select the combination of data mining models, tools and databases that suit our needs.
Four Components or Layers in Data Mining


Thursday, May 31, 2012

Floating Point Representation: the Loss of Precision

On SAS PC, 11169568236203649 is actually represented as 11169568236203648 due to the limitation of  64-bit IEEE floating point representation. 


In most of the applications, there is practically no difference between  11169568236203649  and 11169568236203648. Unfortunately, every digit matters in the case of function mod. 

  

Tuesday, May 29, 2012

A Bug in SAS MOD Function.

MOD( m, n)  is one of the most commonly used functions. It  returns the remainder of m divided by n and the result is deterministic. We would expect it to be calculated correctly by any data analysis software packages.

However, we have found an issue in SAS mod function.The calculation we tested is  mod(11169568236203649, 30269).  SAS returns different results than Oracle and R do. SAS returns 11731. Both Oracle and R return 11732.

SAS script (SAS Version 9.2) and the result.
data test; val=mod(11169568236203649, 30269) ; run;
11731
or
proc sql; select mod(11169568236203649, 30269) from dummy;
11731

Oracle SQL script and the result.
select mod(11169568236203649, 30269) from dual;
11732

R script and the result.
11169568236203649 %% 30269
11732

We have verified that 11732 is the correct answer using two independent approaches. You may try the above scripts on your own computer and let me know what you find out. ( I explain my approaches to verify that 11732 is the correct answer of mod(11169568236203649,30269) and why I bother to calculate mod(x, 30269) in other posts on this blog.)

SAS Is Wrong! R and Oracle Are Right. MOD(11169568236203649, 30269)=11732



For function MOD(11169568236203649, 30269), SAS returns 11731 while both Oracle  and R  return 11732. I used the following two approaches to verify SAS Is Wrong.  Oracle and R are right.
Approach 1. The following two Oracle Queries show that 11732 is the remainder
SQL>  select ( 11169568236203649 -11732)/30269 vl from dual;
        369010150193.000000
SQL>  select ( 11169568236203649 -11731)/30269 vl from dual;
        369010150193.000033


Approach 2. The following PL/SQL scripts.
At first, I wanted to run the following PL/SQL Script 1. It took too long and I had to kill the process. Instead, I ran Script 2 shown below and it returned  11732 at the end of the loop.
PL/SQL Script 1.
set serveroutput on;
declare
remainder number;
begin
remainder:=11169568236203649;
while (remainder >30269)
loop
remainder:=remainder-30269;
end loop;
dbms_output.put_line(remainder);
end;
/
PL/SQL Script 2.
set serveroutput on;
declare
remainder number;
begin
remainder:=11169568236203649;
while (remainder>302690000000)
loop
remainder:=remainder-302690000000; /* equivalent of 10,000,000 loops of -30269 */
end loop;
dbms_output.put_line(remainder);
while (remainder>30269)
loop
remainder:=remainder-30269;
end loop;
dbms_output.put_line(remainder);
end;
/
At the end of the loop, it returns 11732.