<a href="http://datasciencereport.com/2014/02/09/top-5-enterprise-hadoop-stories-of-2013/">February 9, 2014</a> · by <a href="http://datasciencereport.com/author/friars93/">Ted O&#8217;Brien</a> · in <a href="http://datasciencereport.com/category/case-studies/">Case Studies</a>, <a href="http://datasciencereport.com/category/data-resources-tools/">Data Resources &amp; Tools</a>, <a href="http://datasciencereport.com/category/hadoop/">Hadoop</a>, <a href="http://datasciencereport.com/category/news-articles/">News Articles</a>, <a href="http://datasciencereport.com/category/top-ranked/">Top Ranked</a>. ·



By: <a href="http://searchdatamanagement.techtarget.com/contributor/Jack-Vaughan">Jack Vaughan</a> : Jack Vaughan is SearchDataManagement’s news and site editor. Email him at <a href="mailto:jvaughan@techtarget.com">jvaughan@techtarget.com</a>, and follow them on Twitter: <a href="https://twitter.com/sDataManagement">@sDataManagement</a>.&nbsp; <a href="http://searchdatamanagement.techtarget.com/feature/Top-five-enterprise-Hadoop-stories-of-2013?asrc=EM_NLN_25941240&amp;utm_medium=EM&amp;utm_source=NLN&amp;utm_campaign=20140102_Top%20five%20enterprise%20Hadoop%20stories%20of%202013_mewebb&amp;track=NL-1816&amp;ad=891000">Original Article Source</a>



<figure class="wp-block-image"><img decoding="async" src="https://lh7-us.googleusercontent.com/-dmYGsE2iNwHqN5j0-PkWLUv6tqtbyChCmcZZZDrhYyNICZwey-vMeTJxJAXCbSM3vn-isZXriwWSaYLlVgnlCfadBAFva3vjCS8zZazJkUthb_AUsii6KopNMrz1v3hfW25M490bxhoDND5tbV110I" alt="hadoop 2"/></figure>



Jack writes: It was clear when 2013 began that open source Hadoop was entering a new phase. It had moved from its original roots in large-scale, Yahoo-style Web applications and was appearing in analytical pilot projects across a variety of enterprises. During the year, software companies worked to add features to the Hadoop data platform in order to enable its wider use in production. As the spotlight shone on the software often represented by a small elephant, SearchDataManagement endeavored to cut through the hype that can obscure the real trends.



Our editors have reviewed our most popular Hadoop-related <a href="http://searchcio.techtarget.com/news/2240211515/SearchCIOs-top-stories-of-2013-Big-data-cloud-and-more">stories this year</a>, and taken together, they form a narrative of Hadoop in 2013. The content followed the path of Hadoop and related software tools, such as HBase, as they gained footholds in the enterprise. We also saw flurries of product activity, including new Hadoop distributions from major IT vendors. From mid-year to year’s end, a new version of the platform known as Hadoop 2.0 — complete with enterprise enhancements — gained attention.



<a href="http://searchdatamanagement.techtarget.com/tip/Hadoop-helps-bring-big-data-into-a-data-warehouse-environment">Hadoop helps bring big data into a data warehouse</a>. During the year, Hadoop implementers in greater numbers began to place their systems into workflows attached to existing enterprise data warehouses. The effect has been particularly noted on established extract, transform and load, or ETL, architectures. Hadoop’s ability to stage data with less reliance on full-scale up-front schemas has in some cases been a plus.



See these related stories: <a href="http://searchdatamanagement.techtarget.com/news/2240178820/Confronting-MapReduce-Hadoop-problems-and-complexities">Confronting MapReduce, Hadoop complexities </a><a href="http://searchdatamanagement.techtarget.com/news/2240182009/Up-from-the-sandbox-Hadoop-data-management-rising-in-importance">Hadoop’s move up from the developer’s sandbox </a><a href="http://searchdatamanagement.techtarget.com/news/2240184329/Expanded-Hadoop-use-cases-will-drive-need-for-new-enterprise-features">Expanded Hadoop use cases will drive need for new enterprise features</a>



<a href="http://searchdatamanagement.techtarget.com/feature/Big-data-fast-Avoiding-Hadoop-performance-bottlenecks">Big data, fast: Avoiding Hadoop performance bottlenecks</a>. Experience with Hadoop in the field shows that more than just “some assembly is required.” In very many shops, Hadoop needs tweaking and enhancements to meet enterprise needs. Our reporting also indicated the Hadoop-style of data processing that worked well at Google and Yahoo may not be the cure for every company’s problem.



See these related stories: <a href="http://searchdatamanagement.techtarget.com/opinion/Googles-big-data-infrastructure-Dont-try-this-at-home">Google’s big data infrastructure: Don’t try this at home </a><a href="http://searchdatamanagement.techtarget.com/video/White-Mind-the-hype-in-evaluating-and-choosing-Hadoop-technology">Mind the hype in choosing Hadoop technology</a>



<a href="http://searchdatamanagement.techtarget.com/news/2240179304/EMC-Intel-unveil-new-Hadoop-distributions-but-how-many-is-too-many">EMC, Intel unveil new Hadoop distributions, but how many is too many?</a> If Hadoop was wanting in some areas, there was no shortage of vendors ready to fill in with product improvements. Notably, the year witnessed IT heavyweights EMC and Intel entering the Hadoop Derby. Easier configuration was often a hallmark of the product enhancements.



See this related story: <a href="http://searchdatamanagement.techtarget.com/news/2240187351/Evolving-Hadoop-ecosystem-presents-new-ways-to-program-big-data-apps">Evolving Hadoop ecosystem presents new ways to program big data apps</a>



<a href="http://searchdatamanagement.techtarget.com/news/2240187351/Evolving-Hadoop-ecosystem-presents-new-ways-to-program-big-data-apps">Enterprise Hadoop will need to work with existing processes</a>. In June, the Hadoop Summit in San Jose, Calif., was a coming-out party of sorts for Hadoop 2.0. Spotlighted in this new version is YARN (for Yet Another Resource Negotiator), whose offbeat name belies a significant upgrade that expands Hadoop’s application into undertakings formerly limited to batch processing schemes. This and other bells, whistles and add-ons further targeted Hadoop for use in the enterprise. New features bring new capabilities but also new challenges.



See these related stories: <a href="http://searchdatamanagement.techtarget.com/feature/Hadoop-2-release-adds-potential-uses-and-new-issues-to-consider">Hadoop 2 release adds new issues to consider </a><a href="http://searchdatamanagement.techtarget.com/feature/Big-data-applications-require-new-thinking-on-data-integration">Big data applications require new thinking on data integration </a><a href="http://searchdatamanagement.techtarget.com/podcast/Hadoop-Summit-2013-Where-is-Apache-Hadoop-heading">Where is Apache Hadoop heading?</a>



<a href="http://searchdatamanagement.techtarget.com/opinion/Security-services-company-uses-MapR-HBase-to-calm-data-downpour">Security services company uses HBase to calm data downpour</a>. User experiences show that Hadoop’s use can go beyond analytics to include operations. For Omaha, Neb.-based managed security provider Solutionary Inc., Hadoop did a bit of both. As described by the company’s software engineering director, Hadoop and its compatriot HBase columnar database proved to be a worthy alternative to the ever-expanding use of Oracle Database RAC. For another company, Hadoop and HBase were seen as high-horsepower open source alternatives to a proprietary rules-based system.



See this related story: <a href="http://searchdatamanagement.techtarget.com/feature/Ancestrycom-teams-work-together-to-use-Hadoop-framework-for-DNA-app">Ancestry.com teams work together to use Hadoop framework for DNA app</a>



Jack Vaughan is SearchDataManagement’s news and site editor. Email him at <a href="mailto:jvaughan@techtarget.com">jvaughan@techtarget.com</a>, and follow them on Twitter: <a href="https://twitter.com/sDataManagement">@sDataManagement</a>.

Top 5 | Enterprise Hadoop Stories of 2013

You are excited. You have got that much awaited interview call for that dream analytics job. You are confident you will be perfect for the job. Now all that remains is convincing the interviewer. Don’t you wish you knew what kind of questions they are going to be ask?



As co-founder and one of the chief trainers at Jigsaw Academy, an online analytics training institute, I regularly get calls from our students days before their scheduled interview asking me just this. I am going to share with you just what I share with them. Here you go. Below are a few of the more popular questions you could get asked and the corresponding answers in a nutshell.&nbsp;



Question 1. Can you outline the various steps in an analytics project?



Broadly speaking these are the steps. Of course these may vary slightly depending on the type of problem, data, tools available etc.



1. Problem definition – The first step is to of course understand the business problem. What is the problem you are trying to solve – what is the business context? Very often however your client may also just give you a whole lot of data and ask you to do something with it. In such a case you would need to take a more exploratory look at the data. Nevertheless if the client has a specific problem that needs to be tackled, then then first step is to clearly define and understand the problem. You will then need to convert the business problem into an analytics problem. I other words you need to understand exactly what you are going to predict with the model you build. There is no point in building a fabulous model, only to realise later that what it is predicting is not exactly what the business needs.



2. Data Exploration – Once you have the problem defined, the next step is to explore the data and become more familiar with it. This is especially important when dealing with a completely new data set.



3. Data Preparation – Now that you have a good understanding of the data, you will need to prepare it for modelling. You will identify and treat missing values, detect outliers, transform variables, create binary variables if required and so on. This stage is very influenced by the modelling technique you will use at the next stage.&nbsp; For example, regression involves a fair amount of data preparation, but decision trees may need less prep whereas clustering requires a whole different kind of prep as compared to other techniques.



4. Modelling – Once the data is prepared, you can begin modelling. This is usually an iterative process where you run a model, evaluate the results, tweak your approach, run another model, evaluate the results, re-tweak and so on….. You go on doing this until you come up with a model you are satisfied with or what you feel is the best possible result with the given data.



5. Validation – The final model (or maybe the best 2-3 models) should then be put through the validation process. In this process, you test the model using completely new data set i.e. data that was not used to build the model. This process ensures that your model is a good model in general and not just a very good model for the specific data earlier used (Technically, this is called avoiding over fitting)



6. Implementation and tracking – The final model is chosen after the validation. Then you start implementing the model and tracking the results. You need to track results to see the performance of the model over time. In general, the accuracy of a model goes down over time. How much time will really depend on the variables – how dynamic or static they are, and the general environment – how static or dynamic that is.



Question 2. &nbsp; What do you do in data exploration?



Data exploration is done to become familiar with the data. This step is especially important when dealing with new data. There are a number of things you will want to do in this step –



a.&nbsp; &nbsp; &nbsp; &nbsp; What is there in the data – look at the list of all the variables in the data set. Understand the meaning of each variable using the data dictionary. Go back to the business for more information in case of any confusion.



b.&nbsp; &nbsp; &nbsp; &nbsp; How much data is there – look at the volume of the data (how many records), look at the time frame of the data (last 3 months, last 6 months etc.)



c. &nbsp; &nbsp; &nbsp; &nbsp; Quality of the data – how much missing information, quality of data in each variable. Are all fields usable? If a field has data for only 10% of the observations, then maybe that field is not usable etc.



d.&nbsp; &nbsp; &nbsp; &nbsp; You will also identify some important variables and may do a deeper investigation of these. Like looking at averages, min and max values, maybe 10th and 90th percentile as well…



e.&nbsp; &nbsp; &nbsp; &nbsp; You may also identify fields that you need to transform in the data prep stage.



&nbsp;Question 3: What do you do in data preparation?



In data preparation, you will prepare the data for the next stage i.e. the modelling stage. What you do here is influenced by the choice of technique you use in the next stage.



But some things are done in most cases – example identifying missing values and treating them, identifying outlier values (unusual values) and treating them, transforming variables, creating binary variables if required etc,



This is the stage where you will partition the data as well. i.e create training data (to do modelling) and validation (to do validation).



&nbsp;Question 4: How will you treat missing values?



The first step is to identify variables with missing values. Assess the extent of missing values. Is there a pattern in missing values? If yes, try and identify the pattern. It may lead to interesting insights.



If no pattern, then we can either ignore missing values (SAS will not use any observation with missing data) or impute the missing values.



Simple imputation – substitute with mean or median values



OR



Case wise imputation –for example, if we have missing values in the income field.



Question 5: How will you treat outlier values?



You can identify outliers using graphical analysis and univariate analysis. If there are only a few outliers, you can assess them individually. If there are many, you may want to substitute the outlier values with the 1stpercentile or the 99th percentile values.



If there is a lot of data, you may decide to ignore records with outliers.



Not all extreme values are outliers. Not all outliers are extreme values.



Question 6: How do you assess the results of a logistic regression analysis?



You can use different methods to assess how good a logistic model is.



a. Concordance – This tells you about the ability of the model to discriminate between the event happening and not happening.



b. Lift – It helps you assess how much better the model is compared to random selection.



c. Classification matrix – helps you look at the false positives and true negatives.



Some other general questions you will most likely be asked:



<ul>
<li>What have you done to improve your data analytics knowledge in the past year?</li>



<li>What are your career goals?</li>



<li>Why do you want a career in data analytics?</li>
</ul>



The answers to these questions will have to be unique to the person answering it. The key is to show confidence and give well thought out answers that demonstrate you are knowledgeable about the industry and have the conviction to work hard and excel as a data analyst.



About Sarita Digumarti: Sarita has over 10 years of extensive analytics and consulting experience across diverse domains including retail, health-care and financial services. She has worked in both India and the US, helping clients tackle complex business problems by applying analytical techniques. She has a Master’s degree in Quantitative Economics, from Tufts University, Boston, and a PG Diploma in Management from T.A. Pai Management Institute, Manipal. Sarita’s <a href="http://in.linkedin.com/in/saritadigumarti">Linkedin profile</a>

Published by John

John

Author

Expertise

Social Media