How search can solve big data problems

Gigaom

There are many solutions for figuring out how to parse large amounts of data, but LucidWorks CTO Grant Ingersoll has a suggestion: use search. At GigaOM’s Structure:Data conference in New York City Thursday, Ingersoll laid out his case for why search is a big part of dealing with databases and indexes.

“Search should be a critical part of your architecture,” he told attendees. It is a system building block for any large problem you’re trying to solve that requires a ranked set of results. And it doesn’t have to be just text search, it can be for any type of search, he said.

Thinking beyond traditional search features, like keyword search, will help businesses solve those problems more easily too. And it lets organizations bring in many differing kinds of data sources and more effectively combine them.

And organizations that keep records of how people are using search to access…

View original post 2,595 more words

Advertisements

5 ways big data is going to blow your mind and change your world

Gigaom

Some people say big data is wallowing in the trough of disillusionment, but that’s a limited worldview. If you only look at it like an IT issue it might be easy to see big data as little more than business intelligence on steroids. If you only see data science as a means to serving better ads, it might be easy to ask yourself what all the fuss is about.

If you’re like me, though, all you see are the bright lights ahead. They might be some sort of data nirvana, or they might be a privacy-destroying 18-wheeler bearing down on us. They might be both. But we’re going to find out, and we’re we’re going to find out sooner rather than later.

This is because there are small pockets of technologists who are letting their imaginations lead the way. In a suddenly cliché way of saying it, they’re aiming for 10x…

View original post 843 more words

Big data needs people, leaders and real-time analytics: A Structure:Data 2013 recap

Gigaom

In the afterglow of GigaOM’s Structure:Data conference this week, a few big-picture trends and surprising quotes stuck with us. (Check out our live coverage here.)

Data needs people, my friend

Despite the much-discussed power of data, there are roles for people to play in big data projects. Data increasingly influences companies’ decision making processes, but several speakers hit on the notion that people should be involved in big data storage and analysis.

It all starts with a human question. Before machines generate answers, employees from many departments should feel empowered to ask good questions of data, said John Sotham, vice president of finance at BuildDirect.

Beyond questions, humans need to decide which algorithms to employ and which data to use to answer questions, said Scott Brave, founder and chief technology officer of Baynote.

In data science, machine use algorithms to make decisions with clean data for the sake…

View original post 632 more words

If you think big data is big now, just wait for the internet of things

Gigaom

When the internet of things comes to fruition, the economics of big data analytics will start to pay off, said Gaurav Dhillon, founder and CEO of SnapLogic. It will be a world in which sensored devices will grow far beyond the NEST thermostats and Fitbit devices some of us now have.

The foundation of the current  data warehouse business was the simple bar code, used to replenish inventory. Data warehouses are now a $30 billion business, he said. Now imagine when there’s input from all these other devices and machines generating and transmitting data. That’s when the “numbers of big data start to ring true,” Dhillon said at GigaOM’s Structure Data conference on Wednesday.

There are already apps that ping a retailer if they notice a shopper wandering aimlessly around a superstore. The app will will send a sales associate to help the wayward shopper find what she’s looking for. “That could be quite useful,” Dhillon…

View original post 3,597 more words

EMC’s Paul Maritz: it takes leadership to move companies toward a data-driven future

Gigaom

It would be easy to think that moving companies toward a future in data would be a technological challenge, but Paul Maritz, EMC’s chief strategist and a key member of its Pivotal Initiative spin-off, took a different approach: it’s about human leadership.

Speaking with Om Malik on Wednesday in New York for GigaOM’s Structure:Data conference, Maritz talked about the two potential courses a company might take in heading toward a data-driven future:

“I think there’s two ways. One way is, unfortunately, Darwinian evolutions,” he said. “But others will change and change always comes through leadership, where someone in the organization has to really step up and drive change.”

Maritz said when he was at VMWare, he was fascinated that in looking at different companies in the same businesses, he could see that some would be fifty percent virtualized, and others only ten percent virtualized, simply because of the mindset at the…

View original post 2,336 more words

Another Internet Architectural Guideline

The Connectivist

Another reason for the huge success of Internet and its applications is the following Architectural Guideline:

Avoid vertical integration of network functions as much as possible, since that would introduce dependencies (which suppliers might like to keep out competitors) and obstacles to further growth and changes”. Stay free to choose better components later and interchange parts of the structure without too much influence on the rest. During actual use the vertical components on different layers are connected.

When you drive a car the rubber of your tires meets the road. The road allows many different types of cars now and in the future. Only with a horizontally layered architecture, where the layers (grouped into levels) are agnostic of each other, is is possible to cope with sustained renewal of components with different timescales and lifecycles of innovations and improvements. For example making more lanes on the road has a…

View original post 99 more words

The Internet has its 30th Birthday, Hip Hip

The Connectivist

Foto Internet 2013-03-13 om 19.04.02

 

The MOTOR of our Society = INTERCONNECTION, and one of its main components and facilitators for that, is the worldwide “Internet”, which is getting more important every day.

Internet World

Its birth is fixed by several sources around 1983 *), although its fathers and mothers are many, (like every success has). Those who established its preconditions can be traced back to the late 60’s: Paul Baran, Louis Pouzin, Donald W. Davies and Derek L.A. Barber, Leonard Kleinrock. They where designing and building the first computer-networks. Now 30 years later the “Network of Networks” has about 2.5 billion very active users. The following chart shows how many percent of the total of users each country has, against time.

worldwide-internet-users_50c1f5f869b1d_w1755

The Internet has been and will be a spectacular success, and its growth is still viral ((exponential growth in numbers of nodes, addresses, traffic and teledensities of users; the sustained doubling time W [internet]…

View original post 1,475 more words