Hi Everyone,&nbsp; I’ve seen a few great lists lately of Open Source Tools for Big Data.&nbsp; So I thought I would share the best of what I’ve seen and use a little crowdsourcing from readers to see what’s missing and create a UPDATED master list.



<h3 class="wp-block-heading">Here is a very helpful landscape style visual of the Open Source Tools from the blog:  Big Data Start-ups</h3>



<figure class="wp-block-image"><img decoding="async" src="https://lh7-us.googleusercontent.com/vS9TfVbZMy7iqo-XUMXiEPITdHAEsBYvRJsXHeICIeK_IBu7RYfGeIVmSY2u4FWyrUPsJg1DX7tpWqdbBMZpY6P8lfX8UBmYsXpMMdj9zlrkZpC4hO1GzjZVxxcoHU4Yzkf1yCVwU2R2_zuIg2zBLg4" alt="Open Source Tools"/></figure>



Next we have another list that looks pretty solid from By Fari Payandeh at <a href="http://bigdatastudio.com/2013/09/01/the-best-of-open-source-for-big-data/">Blog: Big Data Studio</a>



Fari writes: It was not easy to select a few out of many Open Source projects. My objective was to choose the ones that fit Big Data’s needs most. What has changed in the world of Open Source is that the big players have become stakeholders; IBM’s alliance with Cloud Foundry, Microsoft providing a development platform for Hadoop, Dell’s Open Stack-Powered Cloud Solution, VMware and EMC partnering on Cloud, Oracle releasing its NoSql database as Open Source.



<figure class="wp-block-image"><img decoding="async" src="https://lh7-us.googleusercontent.com/7sZLzBPLaZIGcCAQLryW4FM4Ob3NwGhp5i9oY_cuieyOovCuoIzIrMZt_fYW8g3yRl9pBcyGlw7n_ZkT8il83PjfYJz5cLnKNkuc5nlmBMIHLIpcpb2a3QAVmJGc3vw4yIPZv-2zZOJOPsM0cHUWDe4" alt="bigdata-opensource-final5"/></figure>



The Final List comes from Datamation.com:



<h3 class="wp-block-heading">50 Top Open Source Tools for Big Data</h3>



1. <a href="http://hadoop.apache.org/">Hadoop</a>



You simply can’t talk about big data without mentioning Hadoop. The Apache distributed data processing software is so pervasive that often the terms “Hadoop” and “big data” are used synonymously. The Apache Foundation also sponsors a number of related projects that extend the capabilities of Hadoop, and many of them are mentioned below. In addition, numerous vendors offer supported versions of Hadoop and related technologies. Operating System: Windows, Linux, OS X.



2. <a href="http://hadoop.apache.org/mapreduce/">MapReduce</a>



Originally developed by Google, the MapReduce website describe it as “a programming model and software framework for writing applications that rapidly process vast amounts of data in parallel on large clusters of compute nodes.” It’s used by Hadoop, as well as many other data processing applications. Operating System: OS Independent.



3. <a href="http://www.gridgain.com/">GridGain</a>



GridGrain offers an alternative to Hadoop’s MapReduce that is compatible with the Hadoop Distributed File System. It offers in-memory processing for fast analysis of real-time data. You can download the open source version from GitHub or purchase a commercially supported version from the link above. Operating System: Windows, Linux, OS X.



4. <a href="http://hpccsystems.com/">HPCC</a>



Developed by LexisNexis Risk Solutions, HPCC is short for “high performance computing cluster.” It claims to offer superior performance to Hadoop. Both free community versions and paid enterprise versions are available. Operating System: Linux.



5. <a href="https://github.com/nathanmarz/storm#readme">Storm</a>



Now owned by Twitter, Storm offers distributed real-time computation capabilities and is often described as the “Hadoop of realtime.” It’s highly scalable, robust, fault-tolerant and works with nearly all programming languages. Operating System: Linux.



<h3 class="wp-block-heading">Databases/Data Warehouses</h3>



6. <a href="http://cassandra.apache.org/">Cassandra</a>



Originally developed by Facebook, this NoSQL database is now managed by the Apache Foundation. It’s used by many organizations with large, active datasets, including Netflix, Twitter, Urban Airship, Constant Contact, Reddit, Cisco and Digg. Commercial support and services are available through <a href="http://wiki.apache.org/cassandra/ThirdPartySupport">third-party vendors.</a> Operating System: OS Independent.



7. <a href="http://hbase.apache.org/">HBase</a>



Another Apache project, HBase is the non-relational data store for Hadoop. Features include linear and modular scalability, strictly consistent reads and writes, automatic failover support and much more. Operating System: OS Independent.



8. <a href="http://www.mongodb.org/">MongoDB</a>



MongoDB was designed to support humongous databases. It’s a NoSQL database with document-oriented storage, full index support, replication and high availability, and more. Commercial support is available through <a href="https://web.archive.org/web/20160216225100/http://www.10gen.com/subscription">10gen</a>. Operating system: Windows, Linux, OS X, Solaris.



9. <a href="http://neo4j.org/">Neo4j</a>



The “world’s leading graph database,” Neo4j boasts performance improvements up to 1000x or more versus relational databases. Interested organizations can purchase advanced or enterprise versions from<a href="https://web.archive.org/web/20160216225100/http://neotechnology.com/">Neo Technology</a>. Operating System: Windows, Linux.



10. <a href="http://couchdb.apache.org/">CouchDB</a>



Designed for the Web, CouchDB stores data in JSON documents that you can access via the Web or or query using JavaScript. It offers distributed scaling with fault-tolerant storage. Operating system: Windows, Linux, OS X, Android.



11. <a href="http://www.orientdb.org/index.htm">OrientDB</a>



This NoSQL database can store up to 150,000 documents per second and can load graphs in just milliseconds. It combines the flexibility of document databases with the power of graph databases, while supporting features such as ACID transactions, fast indexes, native and SQL queries, and JSON import and export. Operating system: OS Independent.



12. <a href="http://code.google.com/p/terrastore/">Terrastore</a>



Based on Terracotta, Terrastore boasts “advanced scalability and elasticity features without sacrificing consistency.” It supports custom data partitioning, event processing, push-down predicates, range queries, map/reduce querying and processing and server-side update functions. Operating System: OS Independent.



13. <a href="https://github.com/twitter/flockdb">FlockDB</a>



Best known as Twitter’s database, FlockDB was designed to store social graphs (i.e., who is following whom and who is blocking whom). It offers horizontal scaling and very fast reads and writes. Operating System: OS Independent.



14. <a href="http://hibari.github.com/hibari-doc/">Hibari</a>



Used by many telecom companies, Hibari is a key-value, big data store with strong consistency, high availability and fast performance. Support is available through <a href="http://www.geminimobile.com/">Gemini Mobile</a>. Operating System: OS Independent.



15. <a href="http://wiki.basho.com/Riak.html">Riak</a>



Riak humbly claims to be “the most powerful open-source, distributed database you’ll ever put into production.” Users include Comcast, Yammer, Voxer, Boeing, SEOMoz, Joyent, Kiip.me, DotCloud, Formspring, the Danish Government and many others. Operating System: Linux, OS X.



16. <a href="http://hypertable.org/">Hypertable</a>



This NoSQL database offers efficiency and fast performance that result in cost savings versus similar databases. The code is 100 percent open source, but paid support is available. Operating System: Linux, OS X.



<a href="http://www.datamation.com/data-center/50-top-open-source-tools-for-big-data-2.html">Click Here to See #17 to&nbsp; 50</a>

50+ Open Source Tools for Big Data

Michael Berry is taking a stand against the big data hype. More data, said the analytics director for travel website TripAdvisor, doesn’t always mean better business results. Case in point: big data and <a href="http://searchcrm.techtarget.com/definition/predictive-analytics">predictive analytics</a>.



“Many predictive analytics applications turn out not to need all of the data,” Berry said during his keynote talk at Predictive Analytics World. So the real task for <a href="http://searchcio.techtarget.com/video/HMS-CIO-Were-big-data-scientists-not-big-data-practitioners">data scientists</a> et al. isn’t figuring out how to analyze all the available data; instead, it’s figuring out how much data it takes to see something worth noting. The bad news?



“There’s not a simple answer to that question,” Berry said.



However, testing the predictive model’s performance by incrementally adding more data can shed light on when enough is enough. For example, when Berry wanted to know the standard bid by travel agency partners for a specific hotel and specific customer, he began computing averages: The first two bids compared to the first three bids compared to the first four bids and so on until he hit a steady plateau at 100,000. If he kept going to 200,000 bids, the average would change, sure, but not enough to matter.



“That’s the way data tends to be: When you have enough of it, having more doesn’t really make much difference,” he said.



So if more data doesn’t matter, what does? “So many things,” Berry said. Working with <a href="http://searchdatamanagement.techtarget.com/feature/Business-data-quality-measures-need-to-reach-a-higher-plane">clean data</a>, doing unbiased sampling, hiring staff dedicated to data quality and creative thinking.



That’s right, there’s a big place in predictive analyses for those <a href="http://searchbusinessanalytics.techtarget.com/news/2240181721/Data-science-team-building-101-Cross-functional-talent-key-to-success">soft data science skills</a>, such as figuring out what variables can make the model stronger or what new patterns might be discovered by combining different kinds of data together. Examples?



“Someone had to think of the idea of wind chill factor,” Berry said, before combining actual temperature and wind speed to reveal a new data point: What the weather will actually feel like.



<h3 class="wp-block-heading">More big data delusions</h3>



Berry wasn’t the only presenter who badmouthed the state of <a href="http://searchbusinessanalytics.techtarget.com/news/2240100743/Predictive-analytics-and-big-data-The-good-the-bad-and-the-ugly">big data and predictive analytics</a>. Karl Rexer, founder of the consulting firm Rexer Analytics, went so far as to suggest that the current crop of data scientists suffers from a bit of delusional thinking.



In his 2013 Data Miner Survey, respondents indicated that the size of data sets is getting bigger. But when Rexer asked them how many records are in a typical data set they use for analyses, “We get the same answer we got in 2007,” he said.



That’s not to say big data is a farce or to give short shrift to the <a href="http://searchcio.techtarget.com/news/2240178592/Hadoop-framework-breathes-new-life-into-Ancestrycom-legacy-tools">interesting work some are doing</a> in this space, he said. “But for the typical analytic predictive modeling/data mining/whatever-you-want-to-call-it project, I would say the overall sample size used for those data mining projects is not increasing.”



<h3 class="wp-block-heading">Name that acronym</h3>



<a href="http://searchcio.techtarget.com/news/2240186181/Learning-native-tongue-of-your-business-is-key-to-reviving-role-of-IT">Translating the language</a> of analytics into something the business can understand is challenging. One way Paychex, a payroll, human resources and benefits service provider, deals with the language barrier is by, well, using language the business suggests.



<h4 class="wp-block-heading">PREVIOUSLY ON THE DATA MILL</h4>



‘Cookie stuffing&#8217;: A <a href="http://searchcio.techtarget.com/opinion/Cookie-stuffing-A-data-scientist-takes-on-seamy-side-of-online-ads">data scientist</a> tackles sleazy side of online ads



Do businesses have the patience for good <a href="http://searchcio.techtarget.com/opinion/Does-business-have-the-patience-for-data-science">data science</a>?



How semi-structured data drives <a href="http://searchcio.techtarget.com/opinion/Semi-structured-data-is-king-of-LinkedIns-recommendation-engine">LinkedIn analytics</a>



“When we build a model, we’ll run a naming contest for the users,” Tom Kern, a risk-modeling analyst for Paychex, said at Predictive Analytics World. Kern’s department will send users an email with a short description about the model and suggest a couple of words to get them started. The users have to come up with “an acronymic name,” he said. So there’s SAM, the sales anticipation model, and TIM, the territory identification and mapping model. “Still working on a TOM,” he quipped.



If the business users’ suggestion is chosen, they get a gift card, and the company gets its users, such as the sales staff, to think about what the predictive model really does.



<h3 class="wp-block-heading">The Tide turns</h3>



The Procter &amp; Gamble Co., one of the biggest consumer goods retailers in the world, announced plans to release a lower-priced version of the laundry detergent Tide in an effort to attract mid-tier customers. Bold move or bad decision?



“One of the big concerns? If you launch products like these, not only are you going to attract folks you currently don’t have, but you’re going to encourage consumers to trade down,” said Shel Smith, partner and founder of <a href="http://searchcio.techtarget.com/news/2240204410/Zipcar-CMO-taps-data-driven-marketing-to-personalize-the-business">marketing analytics</a> firm Twenty-Ten Inc.



That’s especially true after a recession that forced so many consumers to be price-conscious. But Smith, for one, has faith in P&amp;G’s strategy. He believes the company will use predictive modeling, lots of data and highly targeted marketing to make new customers but keep the old.



“There’s something they know that we don’t in terms of their ability to maintain the existing franchise, but go after new consumers by being far more surgical,” he said.

MACHINELEARNING