The topic of this essay explores what the greatest data management challenges are for the private/corporate sector at present and in the near future, with a focus on digital asset management (DAM) within companies that require and/or produce large amounts of unstructured data. The type of companies that need to manage this kind of data can include advertising and digital agencies, television, film, animation, and VFX. While these businesses will have different content, data amounts, and business needs, they all face the ongoing challenge of effectively making the ever-increasing amount of content findable, identifiable, obtainable, preserved, and provide access to digital assets for various purposes.

Data is either structured or unstructured, with structured data organised into schemes of related information in a database and viewed and manipulated using spreadsheet software. Unstructured data, due to its nature, is unable to be organised into a scheme within a database, although its metadata can. Unstructured data may contain photographs, illustrations, audiovisual files, presentations, webpages, and text.

This essay will focus on the internal needs of this corporate sector. Specifically, this essay will focus on the corporate content that is often produced or purchased for the purposes of generating reuse and creating projects for the company. This is a process and system known as Digital Asset Management (DAM) and it allows enterprise-wide content storage and sharing from a central location.  Exploring the topic in this context, this essay will examine the challenges of indexing and classification to create searchable structured data from the unstructured data, which requires the creation of metadata. Next, it will explore the challenges associated with the technical infrastructure and storage needs of this data and an exploration of the strategic business goals. Finally, exploring some of the challenges of making the content accessible and obtainable within the enterprise.

One of the greatest challenges for the identified private sector is the amount of unstructured data produced or managed, and the ability to search, analyse, and provide business users with the ability to leverage the content contained within the data. Approximately 80-90 percent of the world’s data is unstructured, is growing, and is a valuable business asset (Vulture Beat Staff, 2021). With the progress of business digital transformations, the practical challenge of this at present is multifaceted and how businesses address them will affect the business into the future with known and unknown effects. According to a Forbes survey, 95 percent of surveyed businesses have some need to manage unstructured data and 40 percent do so on a regular basis. Unstructured data is critical for the creation of products and services that their customers and clients need, and is vital to remain competitive in their markets (Kulkarni, 2019). Technological solutions like artificial intelligence, machine learning, and data lakes can both help with managing the surge in the amount of content that needs to be described, and creates their own new challenges (Saoudi & Jai-Andaloussi, 2021). Not knowing what content exists or is available to an organisation severely hampers the business’s ability to leverage their data assets. These challenges require strategy to best understand and navigate corporate or business goals, and operational projects that enable a computerised, systems approach that supports the strategic aims (Gordon, 2013, p. 59).

First, the need to index the content so that it is knowable and findable is a consideration. Addressing this challenge can take the form of the traditional archival approach of describing the content as metadata using a schema, or using automated, machine learning methods, or a combination of both methods (Lichtenstein et al., 2014). Either method produces challenges for any enterprise, which includes costs of either labour or the computing power to automate, and accuracy of either method is a challenge that needs to be understood and mitigated (Saoudi & Jai-Andaloussi, 2021). Because unstructured data cannot be searched without this indexing, the data cannot be used to its full potential, or sometimes at all. Describing the content allows the metadata to be a structured surrogate of the unstructured data and allows the content of the data to be findable by humans and machines (International Association of Sound and Audiovisual Archives, n.d.), and therefore usable. When the content is indexed, classified, and tagged it becomes more obtainable for more people across organisations, cuts down on duplication, helps in making better business decisions, and can help with consistency of projects that use and reuse the content. This leads to the need to better understand metadata and how it helps in addressing this problem.

The rapid growth of unstructured data and the need to efficiently index it is has led to technical innovations such as Content-Based Video Retrieval (CBVR) systems. Before the ability to automate video indexing, the only option was an annotation-based method that uses “textual information, attributes, or keyword annotations to represent the video content” (Saoudi & Jai-Andaloussi, 2021). With machine-learning algorithms, indexing “uses visual features, such as motion, texture, shape, or colors, which describe the low-level visual content” (Saoudi & Jai-Andaloussi, 2021). Combining these two methods is the aim of a CBVR system and provides multilayered description, from creators, subjects, and rights to transcripts of speech and complex analysis of the image. These technological solutions for enabling searching and finding content within unstructured data still come with the challenges of accuracy, processing times, and costs to implement, but are contributing to increasing the many ways this content can be found and in ways that were not possible with textural description alone.

The challenge of metadata creation is of finding the right path for each individual business. Factors for metadata creation might include the amount of unstructured data they have, the volume in which it is arriving at the business, and the speed at which the data needs to be described and ready for use. For smaller projects manually describing the content may be a more affordable option. For large projects, automated description might prove more cost-effective and quicker. As an example of the combination approach, the German National Library of Science and Technology (TIB) had a large volume of video to describe for a website portal of scientific information for which they used a combination of manual indexing with automation to extract content metadata from the speech, text, and visual information contained in the video: first the video was ingested and indexed with bibliographic metadata followed by an automated “structural analysis based on shot detection, representative keyframes for a visual index are extracted, followed by optical character recognition, automated speech-to-text transcription, and visual concept detection” (Lichtenstein et al., 2014). This creates a very rich number of points for users to find what they are looking for and can open the usage in ways that bibliographic data alone cannot. Regardless of the volume or velocity and methods used to create it, metadata creates a standard way to describe the content contained within the unstructured data. The use of controlled vocabularies to describe the content can help with creating a simplified and accessible database for easier enterprise-wide usage (International Association of Sound and Audiovisual Archives, n.d.). Businesses that may be used to managing their unstructured data without a common standard or distribution system may find that their solution to finding and obtaining all their assets is within a Digital Asset Management (DAM) system.  

Once the unstructured data is indexed or classified within a DAM system, and searchable via the structured metadata, the next aspect of the challenge is making the content obtainable via the DAM’s integration with policies, tools, and standards that allow the content to be shared, changed, and tracked. Businesses face the challenge of understanding how and when their unstructured data needs to be accessed and with providing the most appropriate technology solutions in aid of this (Amplifi.io, n.d.). Businesses may find that they have different needs for the data at different times and some data may need to be archived and other data available for immediate use, sometimes known as hot and cold data (Sherbak, 2021). This is also where businesses need to factor in the ongoing preservation of this data and ensuring that the files are in the correct format for both use and reuse, and archiving (Digital Preservation Coalition, 2015). The challenge of how best to obtain unstructured data will be approached differently, depending on content type for each business, business size, and the strategic business needs. DAMs have become an industry standard for addressing the need to provide business-wide access to a company’s unstructured data and the aim of a well-functioning DAM is to ensure producers can find the right digital assets for projects while also protecting and securing the assets by creating access restrictions to the system and assets, be that for legal or position-related reasons (Amplifi.io, n.d.). Policies, procedures, and governance are needed for its well-functioning, with business needs embedded into the systems to ensure that metadata, standards, usage, and automation is easily followed (Lawrence, 2014). See Figure 1 for the core elements of a DAM system.

Figure 1

Core DAM

Note: from Benefits of DAM, by Widen, n.d., (https://www.widen.com/resource/digital-asset-management).

Once the unstructured data has made its way into the DAM with its metadata, making the content obtainable company-wide is a complex task. Delivering the right content to the different departments and users requires an understanding of the users and their level of technological ability. This is where access controls are also needed. A common method for keeping the DAM secure but also providing targeted content to the various users is to create an interface with the DAM in the form of a web portal (Keathley, 2014). Figure 2 shows an example of how many different users with different needs might wish to access the DAM. This configuration can also protect the content on the DAM from any unauthorised use or accidental deletions. Custom search tools can also be created on the web portal, as all content is not always needed. This is where a digital asset manager can provide curated content, provide their contact information for any reference services that the user may need, and provide search strategies and faceted search. Faceted search allows navigation of complex asset sets by using controlled vocabularies, which the user will see as drop-down lists (Keathley, 2014). All the work that goes into search tools, indexing, metadata, policy, and various other tools should lead to users effectively and relatively painlessly finding the exact asset they need to fulfil their own project.

Figure 2

A DAM interface with three departments, an agency, a content management system (CMS), and an upload utility on a web site

Note: From Creating and accessing assets, by Keathly, E. 2014, https://learning.oreilly.com/library/view/digital-asset-management/9781430263760/9781430263760_Ch07.xhtml#Sec9

The facets of indexing, metadata, and the technological systems, policies, and tools constitute most of the complex challenges of making the content in unstructured data findable, identifiable, obtainable, and preserved. The overarching business strategy will provide the direction for operational projects that complete these aims, but without these facets the business strategy is sure to fail. A well-managed and functioning DAM system is the best tool to provide the solution to the ongoing challenges of making large amounts of digital content available for business-wide use, and the creation of metadata enables unstructured data to become structured and searchable. For a business that deals with large amounts of unstructured data, applying these facets helps overcome the challenges and creates benefits to the business by saving time, protecting their brand, assets, market share, and eliminates data silos so to allow their assets to be used effectively enterprise wide.