Advertise with

Big data is seeing organisations becoming more mindful of aligning their data management practices with newer paradigms such as data lakes, elastic and cluster computing, and real-time data.
Big data is seeing organisations becoming more mindful of aligning their data management practices with newer paradigms such as data lakes, elastic and cluster computing, and real-time data.

Working towards better data management


By Stefan Kruger and Peter Gaertner, senior Data Architects at Decision Inc. - Sep 17th, 11:31

Big data is seeing organisations becoming more mindful of aligning their data management practices with newer paradigms such as data lakes, elastic and cluster computing, and real-time data. As such, the cloud provides an environment capable of managing the volume and scale required to do so. 

There has been a continued drive to improve the mechanisms for dealing with the processing and analysis of big data volumes, such as machine learning (ML). However, despite these innovations, challenges such as bad or biased ML models are not easily solved and require continuous attention and improvement moving forward.

This is where an effective data management plan (DMP) is essential. It empowers the business stakeholders to consume and model the necessary data and generate business insights from it. A data management plan does not compromise on security and serves to establish a flexible, agile, and forward-thinking environment needed to cater to big data in a digital world.

Regulatory drive

An integral part of the DMP is to ensure the company remains compliant with current regulations. This requires an understanding of how the collected data is stored, used, and analysed within a complex regulatory environment.

Companies must ensure the affected data is anonymised and encrypted as necessary while applying transparent principles on how it is collected, and the use cases it provides for. All this should be applied in a legal framework suited to all relevant legislative guidelines as set out by governments (think SA’s POPI, the GDPR of the EU, and Australia’s CDR).

And while this regulatory context is key, companies who are adapting their DMPs must also be aware of the most impactful trends in data management. Things such as automation and DataOps, the continued hybrid and full cloud migration of organisations, data streaming from Internet of Things devices, and of course the compliance focus.

Subtle differences

Part of this ‘new’ data environment is DataOps. Contrary to popular belief, this is not just DevOps for data. It requires data management solutions to take into consideration key aspects that have traditionally been less prevalent in the data management space.

For example, ubiquitous version control and collaboration. This is necessary to carry the full solution in centralised source control and facilitate collaboration between stakeholders. It often has the nett effect of reducing the reliance on traditionally UI-driven tasks and changing these to automated scripts.

Secondly, DataOps sees infrastructure as code also playing a role. This defines the infrastructure needed by a solution and packaging it as a defined, dynamic, and versionable part of the data management process.

Underpinning these elements is the need to enable automated testing routines that run when a new version update is done as part of the DMP process. Automated feedback and flagging of testing outcomes to the required stakeholders form a core part of this.

Good data

Part of ensuring an effective DMP is put in place comes down to process engineering. Companies must embrace good process and change existing ones where necessary. Decision-makers must avoid the risk of simply digitising or regurgitating bad processes.

Also, it must be remembered that on-premise infrastructure does not necessarily always translate directly into cloud architecture concepts. Companies must ensure that any cloud-bound migration (whether hybrid or full) is designed to leverage the targeted cloud benefits where it makes the most sense and delivers the most impact from a functionality and a cost savings perspective.

Data management will continue to grow in impact as more businesses embark on exercises in digital transformation. The cloud-oriented focus will continue, but the legacy of on-premise solutions will still be around for some time, especially in enterprise environments.

The convergence of solutions catering for data management (such as master data, data quality, integration, storage, and intelligence) will remain an interesting area to watch as technologies evolve. However, a DMP requires a considered approach that factors in both existing and future data requirements. Companies must be willing to embrace change in this new digital world.

Related News

Woolworths carves out market share in SA
27/11/2019 - 10:11
In Australia, David Jones's sales declined 2.1%, with the company saying a store refurbishment contributed to the decline.

Push and pull strategies work together to keep consumers coming back for more
26/11/2019 - 10:20
The retail sector is under increasing pressure as consumers have shrinking disposable income in a strained economy. Maintaining share of wallet is critical. Relying solely on a push route to market strategy from manufacturers into retailers is not enough to get consumers buying products. A pull strategy needs to coexist with the push to drive brand consumption. Integrating these strategies requires intelligent and insightful decision-making. This, in turn, requires data generated through smart technology which provides line of sight across the value chain from manufacturer to distribution, retailer to the consumer.

Today’s customers are loyal to speed and convenience, not brands
25/11/2019 - 11:15
Consumer expectations are rapidly shifting as technologies such as mobile, geolocation, social media and increasingly, Internet of Things devices and wearables, connect people to a world of easily accessible information and convenient services. With the ability to browse, compare and order with a few swipes and taps, consumers are becoming trained to value convenience and service above nearly anything else.

Gearing FMCG manufacturing for the red season spike and maximising profits all year round
25/11/2019 - 11:03
As we enter the festive season, demand for Fast-Moving Consumer Goods (FMCG) increases rapidly, often leaving manufacturers scrambling to fulfill orders from their distribution channel. If demand cannot be met, then loss of revenue is inevitable. However, over-production is not an ideal solution either, as it can leave manufacturers sitting with unsold stock that costs money to store.

Black Friday not necessarily a “black & white” decision for small business growth
25/11/2019 - 10:52
Black Friday, once only a North American marketing frenzy, has become a critical entry in the calendars of South African retail business owners.