LucidLink Filespaces – Cloud-Native File System
LucidLink
Category
Cloud Platform Innovation of the Year
Entry Description
What exactly is novel about your product?

LucidLink is transforming the cloud file services paradigm. LucidLink Filespaces™ has been designed to address the limitations of NFS and SMB protocols. It’s a modern file system that works equally well over LAN and WAN, has a high-performance concurrent streaming architecture, reduced network chatter and a robust security model, and is compatible with all major operating systems (Windows, Linus, macOS).

Filespaces is a cloud-native file system that works with any S3 compliant object storage, optimized to perform over distance like a local disk. Filespaces enhances support for file-based production workloads by providing high-performance file services and streaming remote access for distributed teams. Both users and applications can connect to the same global namespace and use it as a local, shared volume. Data streams on-demand, reducing complexity, optimizing distributed workflows, and enhancing team collaboration and productivity.

LucidLink Filespaces provides a uniform shared, global file system namespace across any device connected to the Internet. It is a SaaS solution, which delivers the intelligence to make any simple object storage act as a high-performance, file system volume shared and accessible from anywhere. Specially developed to offer a near-local user experience, even when accessing data over distance, Filespaces functions like network attached storage (NAS) or file server, but both the clients and data repository can be located anywhere. All authorized client devices can see and access the same shared data set, irrespective of their operating systems or geographic location, as long as they have network access to the S3 object store.

First, each client device runs a software agent that integrates with the local OS to present a file system mount point that appears as a local file system volume. LucidLink’s file system client talks directly to the metadata service and the object store on the backend. There are no intermediary gateways or file servers. The second component, the metadata service, is a discrete process offered by LucidLink as a service, which is entirely transparent to the end-user. It provides file system metadata information to all client nodes—while keeping the metadata separate from the object-store. The third component, the object storage, is the data repository where all file content is stored. File data is first compressed and encrypted on the client device before being stored. Metadata is continuously synchronized between devices, while the data is kept in place and only fetched as needed. Most of the intelligence is built into the client software, which assembles the metadata and the data to present a complete, POSIX-compliant, random-access file system to its host operating environment. The data is not synchronized but instead streamed on-demand to and from each client device. Data streaming is based on the requests, which local applications running on each device make to the file system. LucidLink offers instant access to extensive data sets that are kept in the object store but are immediately available when needed.

Performance is enhanced through metadata synchronization, local on-disk caching, intelligent prefetching, inline compression, to multiple parallel streams.

How does the product break the conventional ideas or processes in the field?

In the age of digital transformation, data grows exponentially, as does the need to access it instantly across locations. It’s a dilemma that the cloud was supposed to solve, but even in the cloud, distributed remote data access can be time-consuming, cumbersome, and duplicative. That’s because up until now, cloud storage and file services have re-packaged technologies initially designed for on-premises operations and run them in the cloud. LucidLink breaks with convention by targeting the source of the problem – distance and latency – thereby allowing customers to consolidate their data in a secure, cloud-based SaaS platform that provides high-performance, streaming data access with no need to download and synchronize files when accessing from remote or globally distributed locations. LucidLink moves beyond backup, archive, and re-factored native S3 applications by transforming object storage into general-use storage, using it as if it were a local, shared volume.

Three unique innovations underpin our high-performance offering and distinguish it from other solutions in the cloud services ecosystem. First, we created a log-structured, cloud-native, distributed file system that extends the value of object storage. By using the object store in a unique data layout – as a block-level, scale-out storage platform – LucidLink can deliver higher performance and infinite capacity. Next, we developed a specific protocol – using parallel TCP streams; local write-back caching; in-line compression; and a proprietary, adaptive algorithm for advanced data prefetching – that solves the problem of distance and latency in cloud environments by removing up to 80 percent of the traffic between applications and files. This protocol separates metadata from file data, synchronizes the metadata locally, and enables high-quality, on-demand data streaming from any location as users or applications request access. Finally, remote distributed workloads don’t exist without concern for data security, so LucidLink took a novel approach to data confidentiality in the cloud – one that minimizes the amount of trust required by all entities involved in storing, managing and transferring data. LucidLink’s “zero-knowledge” encryption model uses robust end-to-end, full-system encryption where all data is encrypted on the customer device and remains encrypted both in transit and at rest with only the customer in possession of the encryption keys – neither LucidLink nor the storage provider (e.g., Amazon S3) can “see” the data. These factors and more put LucidLink at the forefront of cloud platform innovation and enable us to provide seamless, secure, on-demand data access, no matter how distributed or remote the workforce, or how large and dense the data file.

How does it go beyond marginal improvements on something that already exists?

Over ten years ago, Dropbox and Box took a big hammer approach to the problem of remote data access for distributed workforces. The idea was simple, put a copy of the file wherever you need to access it and keep changes synchronized across all devices. This worked for users with small-sized files and data sets with casual sharing requirements, and both companies achieved success. However, as data sets and file sizes grow, this approach doesn’t scale. Users still need to download and sync files, consuming precious time and resources. Cloud vendors attempted to fix the problem with native solutions, but these don’t work for hybrid, multi-cloud, or mixed OS environments. Storage gateway vendors also offered options, but these are expensive, complex, and break down over multiple locations.

By design, LucidLink goes beyond merely making marginal improvements to existing technologies. Specifically, Filespaces addresses the limitations of both the NFS and SMB protocols. It is a modern cloud file service that works equally well over LAN and WAN, has a high-performance concurrent streaming architecture, reduced network chatter, and robust end-to-end encryption (AES-256 in GCM mode) that protects data at rest and in flight. LucidLink does away with synchronization to stream data on-demand directly from the cloud – instantly, regardless of the operating system or device. Filespaces maintains the shared data store in the cloud, with frequently accessed data cached locally to deliver the best of both worlds in terms of performance and flexibility when accessing TB to PB-range volumes. Filespaces untethers devices from their location and allows the same uniform experience, whether in or outside the office, on-premises or in the cloud. LucidLink’s software-only, cloud-first approach significantly expands the use cases for cloud file systems, provides much better integration with the host operating system, and eliminates capital expenditures. The result is the next generation of data center infrastructure that delivers an all-new level of speed, functionality, and cost savings.

How do customers benefit from the product or innovation?

As organizations move toward digital transformation, they seek a file service that is secure, agile, easy-to-manage, scalable, and cost-efficient. LucidLink Filespaces™ delivers this and more. Filespaces is a no-compromise solution for leveraging file services for private or public clouds. It provides unlimited capacity that requires no infrastructure and works with any S3 compliant storage provider, any file type, any device, and all major operating systems. With innovative functionalities such as direct read/write access from the cloud, snapshot support for backup and security, persistent caching and prefetching, user access control, and global file locking, customers benefit from streamlined collaboration and workloads, time and cost savings, reduced to no data duplication, and tighter control over data storage, security, and device and sovereignty mandates.

LucidLink has a number of use cases in M&E and AEC industries and the US Government as well as partnerships MSPs and distributors.

John McNeil Studios (JMS), a California-based creative studio that works with media-rich data files, is already benefiting from the Filespaces platform. JMS cites benefits such as improved remote access to large video files, costs savings via better remote access for freelance workers, and further cost savings because Filespaces works on their existing storage area network (SAN).

A US government agency uses LucidLink for a common data repository for distributed workflows. According to one senior analyst at the agency: “Retrieving file type data from S3 is a nightmare. LucidLink offers an elegant way to get around all that. With LucidLink, I can share things over much further distances which is impossible to do any other way. This is a complete game-changer for the way we handle video. We can never go back.”

LucidLink’s partners – managed service providers and cloud service providers – are also benefiting from the ability to offer a responsive, available, durable cloud-based platform that’s low-cost, high-performing, secure, and flexible enough to support multiple operating systems within the same installation.
Supporting Documents