Advertise with

Big data is seeing organisations becoming more mindful of aligning their data management practices with newer paradigms such as data lakes, elastic and cluster computing, and real-time data.
Big data is seeing organisations becoming more mindful of aligning their data management practices with newer paradigms such as data lakes, elastic and cluster computing, and real-time data.

Working towards better data management


By Stefan Kruger and Peter Gaertner, senior Data Architects at Decision Inc. - Sep 17th, 11:31

Big data is seeing organisations becoming more mindful of aligning their data management practices with newer paradigms such as data lakes, elastic and cluster computing, and real-time data. As such, the cloud provides an environment capable of managing the volume and scale required to do so. 

There has been a continued drive to improve the mechanisms for dealing with the processing and analysis of big data volumes, such as machine learning (ML). However, despite these innovations, challenges such as bad or biased ML models are not easily solved and require continuous attention and improvement moving forward.

This is where an effective data management plan (DMP) is essential. It empowers the business stakeholders to consume and model the necessary data and generate business insights from it. A data management plan does not compromise on security and serves to establish a flexible, agile, and forward-thinking environment needed to cater to big data in a digital world.

Regulatory drive

An integral part of the DMP is to ensure the company remains compliant with current regulations. This requires an understanding of how the collected data is stored, used, and analysed within a complex regulatory environment.

Companies must ensure the affected data is anonymised and encrypted as necessary while applying transparent principles on how it is collected, and the use cases it provides for. All this should be applied in a legal framework suited to all relevant legislative guidelines as set out by governments (think SA’s POPI, the GDPR of the EU, and Australia’s CDR).

And while this regulatory context is key, companies who are adapting their DMPs must also be aware of the most impactful trends in data management. Things such as automation and DataOps, the continued hybrid and full cloud migration of organisations, data streaming from Internet of Things devices, and of course the compliance focus.

Subtle differences

Part of this ‘new’ data environment is DataOps. Contrary to popular belief, this is not just DevOps for data. It requires data management solutions to take into consideration key aspects that have traditionally been less prevalent in the data management space.

For example, ubiquitous version control and collaboration. This is necessary to carry the full solution in centralised source control and facilitate collaboration between stakeholders. It often has the nett effect of reducing the reliance on traditionally UI-driven tasks and changing these to automated scripts.

Secondly, DataOps sees infrastructure as code also playing a role. This defines the infrastructure needed by a solution and packaging it as a defined, dynamic, and versionable part of the data management process.

Underpinning these elements is the need to enable automated testing routines that run when a new version update is done as part of the DMP process. Automated feedback and flagging of testing outcomes to the required stakeholders form a core part of this.

Good data

Part of ensuring an effective DMP is put in place comes down to process engineering. Companies must embrace good process and change existing ones where necessary. Decision-makers must avoid the risk of simply digitising or regurgitating bad processes.

Also, it must be remembered that on-premise infrastructure does not necessarily always translate directly into cloud architecture concepts. Companies must ensure that any cloud-bound migration (whether hybrid or full) is designed to leverage the targeted cloud benefits where it makes the most sense and delivers the most impact from a functionality and a cost savings perspective.

Data management will continue to grow in impact as more businesses embark on exercises in digital transformation. The cloud-oriented focus will continue, but the legacy of on-premise solutions will still be around for some time, especially in enterprise environments.

The convergence of solutions catering for data management (such as master data, data quality, integration, storage, and intelligence) will remain an interesting area to watch as technologies evolve. However, a DMP requires a considered approach that factors in both existing and future data requirements. Companies must be willing to embrace change in this new digital world.

Related News

How to attract and retain the best employees
14/10/2019 - 11:15
Finding the right employee can be a time-consuming process – especially when you’re looking to hire the best candidate for the job. But there are simple steps one can take to find employees who are best suited for your culture and the future success of your business.

Securing your brand is a C-suite challenge
11/10/2019 - 14:37
Brand value is a key component of a company’s success and it takes several years to establish by implementing a brand strategy that supports the business. However, in today’s digital environment, brand value can be compromised in a matter of minutes and negatively affect the brand and company. As such, brand security is an issue that needs constant attention to minimise possible reputational damage to the brand and the associated risk of losing customers.

Understanding workforce management in security
10/10/2019 - 15:08
In an increasingly connected business environment, decision-makers need to more effectively leverage the human resources they have at their disposal. According to dormakaba South Africa, this is where a good workforce management system that can manage personnel as a strategic asset becomes imperative.

Temporary employment services offer a dynamic staffing solution to help the retail sector cater for the seasonal peak
10/10/2019 - 14:56
As we head into the final stretch of the year it is time to start gearing up for the added demands of the summer season. From factories and warehouses to retail outlets, the entire retail chain is under added pressure during this seasonal time of high demand. Temporary employment services (TES) can help all areas of the retail sector to cater to this peak with additional and flexi staff exactly when and where they are needed, with the right skills for the job, and none of the administrative headache of hiring permanent or contract staff.

Business sentiment takes a turn for the better in September
09/10/2019 - 15:34
Business sentiment, as measured by the SA Chamber of Commerce and Industry (Sacci) business confidence index, took a positive turn in September.